I've recently been taking a course on blockchain, as it's something that I find quite interesting.

Blockchains have recently gained popularity in several industries, and although I am yet to use them in my day to day, it's something I can definitely see comin in the near term.

Blockchains can solve problems and create efficiencies in tech areas such as privacy, security and data sharing. Blockchains help to move away from the centralized "one source of truth" monolith database, that most companies have. In fact in almost every job I have had, there is always a huge, single production database, that contains all the business critical data, and if it were to go down, there would be chaos.

Since as far back as I can remember, we have always been trying to move away from centralised computing, and splitting software into smaller chunks, microservices and moving into the cloud. One thing that always remains is a huge monolith database. I've seen this change slightly into smaller, domain specific databases and stores, but usually there is still a single point where everything comes together.

A blockchain is a type of distributed ledge technology, which basically means it is a data structure which resides across multiple computing devices, typically spread across multiple devices and regions. DLTs existed before bitcoin, but bitcoin brought together some of the core ideas, with regards to timestamping, P2P, crytography and sharing the computing power. A good description of a

DTL could be summed up in through points:

- Data structure that captures the current state of a ledger
- Transactions that change the ledger state
- Protocol to allow transactions to be accepted

A blockchain will track various assets transactions. The transactions are are grouped into blocks. There be any number of these transactions in a block. Nodes on the blockchain network group up the transactions and send them through the network. They are synced up by an agreement by all peers, and eventually each node will contain an up to date version of the ledger. Blockchain just basically a P2P distributed ledger, which is ruled by smart contracts. Smart contracts are just predefined actions that are performed when certain conditions are met.

Blockchain used to be referred to as the specific data structure within a DLT, but now a lot of people use the term blockchain to refer to anything, from cryptocurrencies to enterprise deployments of DLTs. Blockchain is a very wide term, and is comparable to the word internet.

Blockchain is in a basic form, a chronological chain of blocks of transactions, that are bundled together and added to the chain, at the same time.

Bitcoin is a good example to explain this in more detail:

Miner nodes bundle unconfirmed and valid transactions into a block. The miners must then solve a cryptographic challenge, to propose the next block. People refer to this as "proof of work".

A block consists of 4 metadata.
- Reference of previous block
- Proof of work (known as nonce)
- Timestamp
- Merkle tree root of transacions

to be continued...
#FrontEndNorthConference 2018

#FrontEndNorthConference 2018

I went for a day trip to sheffield to attend the FrontEndNorthConference

In summary, it looks like we are already doing what the rest of the industry suggests as best practices, although there were some interesting things I learned whilst there.

The schedule went as follows:

Opening Remarks
09:30  -  10:00
FrontEndNorth 2018 begins with opening remarks, including a word from our headline sponsors and details of our fantastic competition prizes.

Night of the Living Style Guides
10:00  -  10:30
The line between design and development is getting blurrier, and our approach to web design is becoming more modular. How do we keep track of it all?

OfflineFirst Apps with PouchDB and CouchDB
10:30  -  11:30
Building an OfflineFirst application means creating something that works fast and well, regardless of whether your device is connected, experiencing a network glitch, or offline completely.

Designing form validation the right way
11:45  -  12:30
For the longest time, I’ve felt frustrated with how client-side form validation worked both technically and more importantly from an end-user perspective.

Reacting to change
12:30  -  13:00
When we have come to learn JavaScript in the past, a large emphasis has been put on understanding the prototypical inheritance model and how you apply object orientated patterns such as encapsulation to a language which lacks a class type.

Agile working in a slow-moving world
14:00  -  14:30
Web development moves fast, and as a digital team we want to deliver the best work we can as quickly as we can. But universities are slow: for every big decision there’s a committee or a steering group to deal with.

Build Anything With React
14:30  -  15:15
This talk explores how we can use familiar tools, React and JavaScript, to build applications across platforms.

Owning Innovation
15:15  -  15:45
Digital innovation is a phrase that’s bandied around a lot, but what does it mean to innovate?

Selling the Front-end
16:00  -  16:30
It might seem easy but selling the concept of how important this ‘Front-end’ thing is to clients is an increasing challenge and shouting out buzzwords and acronyms isn’t helping anyone.

Automating Critical CSS
16:30  -  17:00
Inlining a web page’s critical CSS to optimise usable load time is already established good practice, but taking it from a manually run local script to a continuous integration pipeline can be daunting.

Marty McFly Has A Lot To Answer For
17:00  -  17:45
6 years ago, Craig gave a talk about pointless stuff. 5.5 years ago, Craig gave a talk about Paul Daniels’ approach to web design. After a 5 year hiatus, Craig is back again to talk about something new.

Closing remarks
17:45  -  18:00
Closing remarks including the announcement of the competition winners!

The first talk was all about colour, and styling, I feel like I learned a lot about accessibility, it's not really something  I have had much experience at all in, in my career so far.
A few tools were suggested for accessibility testing, but this one looked good:

The main thing I took away is that it's easy to exclude people who use ScreenReaders and have ColourBlindness by using bad design and colour choices

As a guide, we should have color contrast ratio as 3:1 for non-decorative text, sized larger than 18 point or larger than 14 point if bold. Text smaller than that should meet a contrast ratio of at least 4.5:1.

The speaker also went into detail about using the built in chrome developer accessibility tools checker (which is something I never really looked at before)

She also mentioned the Web Content Accessibilitiy Guidelines, which is something I should probably read up more on.
It has recommendations such as content should be distinguishable and requires we make it easier for users to see and hear content including separating foreground from background.
Another useful tool to use is colourblind filters, which will show what your site looks like to a user who is colourblind.

The talks then shifted towards the importance of using a styleguide for consistency across your site.
A good example of a styleguide is

This looks very similar to our styleguide (although it seems a bit easier to navigate around than our styleguide).

Another cool tool that was mentioned was a style guide generator page, which you can specify a URL and it will dynamically build a simple styleguide, based on the page, although after trying this out, it’s very basic. - Stylify Me was created to help designers quickly gain an overview of the style guide of a site, including colours, fonts, sizing and spacing.

The next speakers was a bit obsessed with zombies!

She created another styleguide template that seemed cool
She spoke about a tool she created, which allows you to export a react application into sketch, which seemed really cool, I need to look more at her github! She also spoke about a style guide template she has used for react dev, which doubles as a react development environment, in a living styleguide
Looks pretty cool

The next speaker talked about the importance of designing sites from an offline perspective. This is a movement that has been taking off recently.

By offline first, we mean we should still be able to use web apps with no internet connection (although I think this is mostly for app developers using a web view, but there are some things we can still take. The speaker also talked a lot about Progressive Web Apps

The speaker uses a tech called PouchDb, which will atomatically sync back with CouchDb (or other compatible noSql database server). She mentioned Couch dB has best replication compared to other databases. Again this looked more related to mobile apps, but I guess you could do something similar on websites.

Basically would mean a user fills something in, or submits some data, and the data is stored (I’m guessing in local storage) until internet connectivity resumes, and then the data is synced with couchDb. She showed an example, which did actually seem to work quite well.

The speaker did talk a lot about databases, and looking around, people started to look a bit bored.

The speaker then spoke about different tools she used for testing http stuff and wrote a pretty good article on it, which include some slides that can be downloaded. - At a conference a few days ago, I put up a slide with a few of my favourite tools on it. I got some brilliant additional recommendations in return from twitter so I thought I’d collect them.

There are a lot of offline first tools, but this one stood out, mostly because of the name - The Offline First Javascript Backend
A fast, simple and self-hosted backend as a service for your (web) apps, Open Source and free. No need to write server-side code or database schemas. Makes building offline-capable software a breeze.

The next speaker focused on why we are doing validation wrong, and about how validation and getting it right is just really hard (which we all know!)
^^ this is a very detailed article he wrote, and its very interesting!!

Web forms don’t have to be irritating, and your inline validation choices don’t have to be based on wild guesses. In his examination of inline form validation options, Luke Wroblewski offers that rarest of beasts: actual data about which things make people smile and which make them want to stab your website with a fork.

He has worked on a project called Conferize form, which he aims to make open source very soon.
It looked cool, but didn’t seem to solve all the issues easily.

A few things we don’t do, but make a lot of sense is
Don't enforce unnecessary rules where the user knows they are right
users know their own name
users know their own Telephone number

I guess this isn’t very practical in a scenario like financial services though!

The next speaker spoke about Reactive Programming, and the stuff this guy writes is really cool

He provides react charts that are super simple to implement and are all open source!
He even has a graph building tool for the CLI, (victory cli)
He wrote a pdf viewer in react, its cool
react-pdf - React component for showing pdf documents

He also created a better webpack cli tool, which looks amazing, and something i'd like to try out in my day to day work. - webpack-dashboard - A CLI dashboard for webpack dev server

The next speaker talked about critical css
He spoke a bit about BEM as well as Atomic Design -
Lullabot - Strategy, Design, Development
BEM & Atomic Design: A CSS Architecture Worth Loving
Using Pattern Lab and BEM on a project I stumbled upon a great way of structuring CSS, better naming, and better a developer experience.
BEM — Block Element Modifier is a methodology, that helps you to achieve reusable components and code sharing in the front-end.

What they aimed to achieve was to get all the critical content, within the first hit from the web server.
Congestion window 14k try to get all content on the very first hit. Prioritise critical content is what Google call it

You can store the fact you got critical css in a cookie so that it doesn't need to Fetch on other pages
Theres a project that helps to automate this called Critical css by filament group
This basically finds the Above the Fold CSS for your page, and outputs it into a file. You then need to look at how you render the rest of the page, which is where it gets messy.

Overall a good conference and great food!!

HacManchester 2017 - GCHQ Winners!

HacManchester 2017 - GCHQ Winners!

Hack Street's Back!

We entered the competition in 2016 and close very close to winning an award, but this year we won!!
The team was made up of me, Dave Mason and Tim Borrowdale.

We decided to take on the GCHQ challenge, because it seemed the most interesting to us, the change was: "to develop a new way to send information to the emergency services. This could involve communicating data in a secure or anonymous way, verifying the received information is genuine or interpreting and analysing information so that it provides new insight."

We came up with a concept of a product called Beacon.

It is a voice-activated device that listened for a particular keyword and notified the emergency services. There would be a web interface for the emergency services to track outstanding calls, and an app that received a push notification when it was triggered and allowed users to cancel the call.

We prototyped the device itself on a Raspberry Pi, running Raspbian.
We had a long-running process that listened for sound starting and ending, then created a WAV file. and uploaded AWS Lex when it captured. We used the speech-to-text functionality to return the sound converted to text, and if we matched on the keyword, we executed a Lambda function (written in node.js) that orchestrated the data storage and the push notification. It sent up meta information also such as the clientip, so that we could do geoip coordination to AWS API Gateway.

The web UI was written using react.js and used API Gateway to get outstanding calls and delete them. The app could also do the same, using the same API Gateway. The web UI is intended to be used as the police HQ for tracking and managing emergency requests.

Apart from some early struggles with the Pi itself, getting all these components talking to each other was actually fairly straight-forward. Lex in particular was very easy to get set up working for our simple use-case, and a shout out to the in-built testing framework in the AWS Lambda console, which made testing / debugging the function much easier than I thought it would be.

We also had to make a video to demonstrate our product. Please keep in mind we were VERY tired, had drank a LOT of coffee (and possibly a couple of beers at the free bar the night before), so cannot be held responsible for the quality or content of the following:

The Awards Ceremony
Having been awake for almost 40 hours, 25 of which were spent frantically coding, what we really needed was... more free food and alcohol! So off we went to the awards ceremony...

Hack Manchester is a great event that the organisers put their heart and soul into every year, and the awards ceremony is always fantastic (particularly if you like cheese jokes), and this years was no exception.

So when we were short-listed for the GCHQ prize, we suddenly felt like it was going to be last year all over again, until the winners were announced... Hack Street's Back! Here we are, collecting our prizes of an Apple TV 4K (64GB) and the much sought after GCHQ puzzle book:

In the day-to-day as a developer, your time is spent (quite rightly) planning your work, writing unit tests, managing CI / CD pipelines, all in the name of quality and stability. Having 25 hours to hack something together (that with a bit of luck will work when you show the judges) is good for the soul. Also, getting outside our usual tech stack and trying out things like Raspberry Pi, node.js and various server-less AWS services was really exciting, and a great eye-opener.

If you're reading this and thinking about entering a team for Hack Manchester 2018, then I'd strongly recommend it. Though be warned, we'll be back defending our title...
Log rotate in unix

Log rotate in unix

Recently I've been doing some work with analytics and tracking and came across log rotation in Linux.

The problem I came across was when parsing nginx logs to build up user journeys. Because the log was set to just keep logging forever, when the logs reached the maximum capacity due to disk size, it occasionally caused nginx to return a 503 response to the caller.

I realised I had two options;

  • Increase the disk size (which would have meant decommissioning the vm in azure, taking it out of the load balancer, resizing and putting it back in) which wasn't really an option as it was a temporary fix.

  • Setup log rotate, so the vm becomes self maintaining (since we are exporting the logs to our analytics platform, keeping more than a few days of logs is pretty pointless, as long as there is sufficient time to reprocess a log if it fails for example)

I was surprised how easy it was to setup!

The Ubuntu VM I was using for nginx needed to have logrotate installed (I just used aptitude) and this automatically added it to cron.daily

cron is configured in  /etc/crontab and its possible to change the time it runs, but I was happy with the default, as long as it was run at roughly the same time daily.

The documentation is pretty good online for this, and within a few minutes I managed to get it all setup and working.
I used a basic configuration to rotate the logs each day, keeping a maximum of 5 daily back ups:

/var/log/nginx/* {
create 0644 admin admin
rotate 5

This means we don't have to worry anymore about the disk running out of space, and our analytics platform can keep on ticking without us having to stop filebeat or having the occasional 503 error being returned from nginx.
AspNetCore Brown bag slides

AspNetCore Brown bag slides

Here are the slides from the brown bag session on aspnetcore that I ran for work

Playing with analytics - Exceptions by page

Playing with analytics - Exceptions by page

Its easy to join tables such as pageViews and exceptions, which will show the amount of exceptions that have occured, on a certain page view, which means you can focus efforts to reduce the amount of unhandled exceptions on pages which get more traffic

| where timestamp > ago(1d)
| join kind=leftouter (
    | where timestamp > ago(1d)
) on operation_Id
| where success == "False"
| summarize exceptionCount=count() by tostring(parseurl(url)["Path"])
| order by exceptionCount desc
| render piechart 

Playing with analytics - Website flows

Playing with analytics - Website flows

I've been playing around with azure app insights analytics again. I focused on seeing if there is a way that i could work out a users journey through a website, using simple query. I know this is easy to do with Google analytics, but I thought I would see if I could do something similar with azure app insights offering.

Turns out its quite simple, and you can see the journey they took, on one device

Mob Programming

Mob Programming

Last night I attended a session on Mob Programming and Working well together, with guest speaker Woody Zuill, who claims to be one of the founders of the practice. The session was at the coop digital building at Federation House in Manchester city centre.

I attended with a few work friends straight after work, and was luckily only a few minutes walk away.
The evening started with networking, and people just getting to know each other before the main talk with a few beers. Woody talked his experience in the industry, and how mob programming came about, and how much more work  an be achieved working in this way, although the velocity cannot be measured so its difficult to prove. It made me realise the small amount of mobbing I have done so far in my career, is probably not as structured as it should be, and certainly not what Woody would consider mobbing!

He showed photos from all around the world, with companies he has converted to use Mob Programming, some interesting photos of companies with 3x80inch screens, looked a bit scary!
He pointed us to a github page where you can run a client that will tell you when the next person is up for being the driver, but this feels a bit too strict in my opinion.

Woody really sold it to us though, the benefits such as the whole squad constantly being up to date with the latest changes, less meetings, quicker answers, shorter feedback loops, squad engagement higher etc. One thing I thought about was that mobbing full on all the time must be really draining, mentally but he summed it up really well with this statement:
Relaxed, Sustainable
Be prepared to contribute
The right thing
At the right time
In the right way

He also spoke about how to work well together, and the basis of this is built with individuals and interactions, kindness, consideration and respect, which seems kind of obvious but it's a nice reference point. He mentioned that being too loud and overpowering takes the light off quieter individuals, and when mobbing or working in a squad, it's important to get the balance right with listening, not over thinking, including quieter members of the squad and having consideration and respect for the other squad members.

I managed to grab some of the slides, which is a good read and took away a quote he left us with which I found interesting

"The value of anothers' experience, is to give us hope. Not to tell us how, or whether to proceed"

Find a good client for Azure Table storage

Find a good client for Azure Table storage

Today I played around with Azure Table storage.

A service I was looking at uses a log4net appender to dump all its logging information, straight into azure table storage. (Log4NetAzureExtensions.Appenders.AzureTableStorageAppender) It has months (maybe years) of data, so running a query was going to take a looooooooooong time.

I must say I've never had experience of using Azure Table storage before, so was unfamiliar with the syntax used for querying results, it looks a bit like powershell but i couldn't seem to limit my result set. I knew that azure table storage is a nosql document store but that's as far as it goes.

Visual studio allows you to connect straight to Azure Table storage, but the UI is really bad, and the query builder even worse. I wanted something quick, where I wouldn't have to spend ages reading some tutorial before i could get something out of it. Guessing at the query language wasn't really getting me far, and each time there was a syntax error, it would wait about 30 seconds before returning an error. 

I did a quick google search, and came across Azure Storage Explorer. The UI was better, but the query builder still seemed pretty poor. It was slow, and didn't really offer anything more than the Visual Studio UI did,

I then found out that Linqpad has support for Azure tables via a plugin.
This meant I could use LINQ to azure table storage. It's pretty fast, and meant it would check my syntax before I ran a query, I really like it!!

If you are stuck for something similar, I most certainly recommend this for quick snippets against Azure Table storage!