Big Data: Is not only a fancy/catchy name

The field of biomedical research has a new trend to use fancy terms in the title of papers/grants in order to attract the attention of reviewers, journals and grant agencies. Amount others are: large-scale, complete map, draft, landscape, deep, full, and Big Data. Figure 1 shows the exponential use of these words in pubmed articles.

Figure 1: Number of mentions of specific terms in pubmed by years.

I will stop here to discuss the term Big data.

Data Visualization: Plots You Should be Using More

Inspired by this blog post

1- Parallel Coordinates — A parallel coordinates graph arrays multiple variables alongside one another with each scaled from highest to the lowest value (highest at the top, lowest at the bottom) and with lines connecting each entity’s position for each variable, horizontally across the graph. Due to a large number of cases represented, it is often presented using an interactive view where individual lines can be selected and highlighted.

Monit: Monitoring your Services

Bioinformatics Applications are moving more in the direction of "Microservices" Architectures where services should be fine-grained and the protocols should be lightweight. Microservices Architectures decomposed the application into different smaller services improving the modularity; making the application easier to develop, deploy and maintain. It also parallelizes development by enabling small autonomous teams to develop, deploy and scale their respective services independently.

With more services (Databases, APIs, Web Applications, Pipelines) more components should be trace, monitor, to know the health of your application. There might be different roles that are played by different services (in different physical/logical machines) that can be even geographically isolated from each other. As a whole, these services/servers might be providing a combined service to the end application. A particular issue or problem on any of the server should not affect the final application behavior and must be found and fixed before the outage happens.

Multiple applications allow developers/devops and sysadmins to monitor all the services in a microservices application, but the most popular ones are Nagios and Monit.