Technical debt is a concept that is often cited and often poorly understood. Many definitions try to oversimplify the meaning of technical debt, limiting to just a handful of causes. However the reality is technical debt accumulates even while doing nothing. Frameworks age and fall out of popularity, industry-standard techniques are replaced or refined, deficits in the original design are discovered, the list goes on. On top of general age and rot, poor communication and planning can leave systems tightly coupled, and not up to standards. Time pressure only adds to the list, including cutting corners, writing poor or limited documentation, and skipping testing. All of these in various capacities contribute to the accumulation of debt over time.
How do you combat this kind of entropic chaos? Some lesser experienced technical leaders might try to pay debt all at once, or even worse not at all. Some project leads call for extreme system decomposition into dozens of microservices. The idea is that each one can be independently rewritten over time, but this introduces a whole host of new engineering complexity. Some try to intersperse improvements as independent tasks, but then which debts get paid and which continue to acrue interest? The truth lies somewhere in the midst of all of these options.
Leaning back into the house analogy, I think there is some good concepts to borrow from the general contracting industry. There are two basic principles that I have seen good builders apply over the years: structure first, and bring things up to code. The first concept is easy - don’t invest in anything relating to finishing until the structure is sound. This means that foundational upgrades such as critical security issues, framework upgrades, and tooling changes should be addressed first. Otherwise any other debt or even improvement tasks that get completed are just begging to be redone. The second concept is driven from a regulatory perspective, but I think applies as well. When a contractor opens up a wall, they need to bring whatever is behind that wall up to code before the inspector signs off on the finished changes. As a result, the estimated cost of bringing things up to code is budgeted as part of the renovation itself. When dealing with a reasonably well built newer house, the renovation cost is likely going to be low. When renovating an ancient DIY house, the contractor is going to be pulling asbestos and lead out of the walls, so it gets planned for with a contingency.
No reasonable individual or contractor would spend thousands of dollars renovating a kitchen in a house with no roof, so why invest in a new jQuery-based theme when a project goal may be to migrate to React or Svelte? Similar to the concept of bringing things up to code, I feel the key to successful technical debt management is to be continuously tackling technical debt. Larger foundational items such as changes in frameworks, languages, and tools should be discussed as major projects, but upgrading and patching should part of a healthy development and maintenance lifecycle. When a service has some major rennovation going on, such as adding new features, the developer should take the time to bring things around that new feature up to code. Migrating from class components to functional components in React? Rarely will it make sense to do that migration all at once, but while a developer is already rewriting and redefining classes in a service? It makes sense to bring that up to code before shipping it off.
I’m not advocating for free-for-all rewrites, every developer for themselves. What I am saying is every feature, every major bug, should take the time to consider current best practices and team standards. If the team decides that classes are just fine and never decide to adopt functional components, that’s not debt, that’s a decision. But a developer going into a module finding a major defect like unsanitized inputs, or a straightforward performance improvement should be empowered to bring the code up to current standards, and document the debt that they paid while working on the original feature.
For me, a great feature of this is the code base is continually improving, no single developer feels burdened doing nothing but paying technical debt, and the cost is negligible. Chances are your developers are paying in some form for debt right now - either quietly making improvements, or being burdened with the need to work around debt, potentially incurring even more debt. When planning we typically ask for an estimate for the task, which often gets padded out quietly. Instead, explicitly build in a contingency for bringing things up to code. An additional advantage is over time, the effort for a task versus the debt paid during that task can be monitored. If the debt ratio is rising, it can be an early indicator that it may be time to look at structual or systemic issues.
In short, these house renovation analogies can fit the software lifecycle well. It offers developers a deeper level of ownership, and it offers an early warning system for uncontrolled debt growth. A software system, much like a house, needs regular and thorough maintenance on its components. Budgeting for these issues, and correcting them early, ensures that the code base remains in peak performance.
]]>In the past few years, we have seen the rapid adoption of artificial intelligence into our daily lives. This has been driven by the development of powerful algorithms, but also by the availability of large datasets to train those algorithms. I believe throughout the next decade this technology will both revolutionize and threaten modern society. Already we are working in symbiosis with intelligent algorithms - from simple autocorrect to complex interactions with voice assistants. As this technology matures, our role will be continuously revised and redefined, in some cases disappearing entirely.
The question is not whether we will accept this change, but rather how we will react to it. I want to believe that the technology we create will be used for the benefit of humanity, but history tells a different story. We are a species that is more than willing to use our tools to cause harm to others. I want to believe that we will be able to control the technology we create, but history tells a different story. We are a species that is more than willing to use our tools to exert our will on others. The question is not whether we will accept this change, but rather how we will react to it. If we react by giving up, we will be left behind. If we react by working together, we will be able to shape the future.
Looking at the work above, it’s incredible to see think that this response was AI generated. With a rich enough initial prompt it is possible to generate large amounts of content, which can then be selectively edited into a cohesive narrative. It could become a powerful tool for generating novel concepts, combining concepts that are linguistically sensible, but perhaps different from typical human thought. This technology will only continue to improve and infiltrate more of our lives, let’s hope that the AI is right and we do work together to shape the future positively for humankind.
]]>Rather than tackle the underlying issue of the divergence of employee value versus compensation, an insidious alternative has appeared over the decade. The rise of the gig economy and side hustle as a means to patch over the compensation issue in our primary jobs is a corporate dream. Workers (and governments) aren’t pushing for fair wages, so many workers take on additional underpaid, high risk work just to make ends meet. The glorification of working culture in North America takes this to toxic levels: a sort of self-imposed 9-9-6 work life. Work your normal 9am to 5pm then go home and work a 5pm to 9pm job, likely also working on the weekends.
Nothing demonstrated this symptom of late-stage capitalism better than the latest commercial from Squarespace, which features a rewrite of Dolly Parton’s 9 To 5. The song is a critical look at the abuse that workers suffer under capitalism. It was then perversely rewritten as a “modern rallying cry for all the dreamers working to turn an after-hours passion or project into a career” and packaged as the solution to feeling underwhelmed and unfulfilled in your primary job. Rather than look at the lack of fulfillment and compensation as a symptoms of a sick system, it encourages to you continue to sacrifice yourself on the altar of capitalismin the name of self-fulfillment.
As the world continues to automate jobs, there will be a growing segment of the working population who will no longer have meaningful jobs. And it seems that the best capitalism has to offer is the opportunity to work soul-crushing bullshit jobs, jobs that provide little value for your employer, and provide little satisfaction for yourself.
Governments could work to correct the inequality in the current system by significantly increasing the minimum wage and indexing it to a realistic inflation portfolio. This would lift a significant number of people who rely on the gig economy to have an acceptable quality of life. The main issue here would be the gradually reducing number of jobs available in a post-automation society. It also continues to rely on individuals, who have a decreasing share of overall wealth, to fund social and societal obligations. Rather, it should be those that hold wealth such as property, and means of production who contribute a larger share into the society from which they reap their benefits.
I think that Universal Basic Income (UBI) is inevitable - people should not have to toil away on meainingless tasks for the right to survive, and we should be able to claim back our days for creative and personal pursuits. It would shift the tax curve to the right, allowing for a typical individual to enjoy a simple condition-free stipend with no tax implications. For those who choose to work, the tax curve would be steeper with the introduction of additional tax brackets. This would promote a healthy common baseline, while allowing individuals to still compete and produce value. It would greatly simplify government program administration as there would be limited need to review, amend, and approve programs.
]]>Beyond the subliminal borderline torture that Facebook encourages, their platforms simultaneously siphon off all of your private data. Whether it was explicit with their Onavo VPN product (delisted due to privacy abuses by Google and Apple), the now-defunct relationship with Cambridge Analytica, or implicit from mining all of your contacts and conversations over the years, Facebook has become a real threat to privacy and safety. Other companies are not blameless, for sure. Amazon is marching down the path of the auth-right, enabling police access to your Ring doorbells around-the-clock in some juristictions, to enabling identification and machine learning en-masse. This is a specific battle I am picking, the battle for my mind. I don’t need the hate and misinformation that Facebook hopes will live rent-free in my mind, encouraging me to come back for another taste.
It’s not going to be easy to leave - and that’s by design. Whenever it came to software I always design with vendor lock-in as a concern, making sure that whatever I write could be ported out. But I didn’t do that with my social life. I am aiming to change that, I will be rebuilding my website and opening up my alternative methods of communication. I will share what I accomplish as I go, and the end goal will be to leave Facebook properties forever.
]]>Once in a while I like to review my subscription choices to see if I am still getting the most value out of each service. Most recently I started paying for YouTube Premium as a way to legitimately skip ands and access exclusive creator content. One perk of this subscription is the inclusion of YouTube music. I have been a happy Spotify subscriber for many years now, but glitchy application behaviour and some missing catalogue items made me second guess whether I am getting the most out of that subscription. I decided to revisit Apple Music and YouTube Music to evaluate their service quality.
Apple Music feels like a quite solid option, with one huge caveat: iTunes. I am a Windows user for my primary desktop and laptop at home and iTunes is an evil that I avoid whenever possible. The Apple experience has continued to decay into near unusability on Windows. I know they have the online player so you can avoid using iTunes, but I wish they would bundle this clean experience into a distinct Windows application. Much of the legacy library and device management that iTunes provided isn’t necessary for my day-to-day music experience. Despite the iTunes nightmare, the experience on my mobile devices such as my iPad and iPhone are excellent.
While there was no direct way to port my Spotify playlists to Apple Music, there was a free online service, Tune My Music, which allowed me to move my Spotify playlists and favourites smoothly to Apple Music. Given these services are all mature at this point I was a little disappointed that this migration process wasn’t native to Spotify, Apple Music, or YouTube Music. Despite the desired locked-in effect, it’d be much easier to poach me as a customer if I could easily take my music library with me.
Fresh off of this mixed Apple experience, I was hoping that Google would have produced something high quality with YouTube Music. They spent months advertising the service to me as an upgrade to Google Music (which I had used prior to Spotify), and a great value-ad to the general YouTube experience. Immediately I noticed how rushed this product felt. Like Apple, they have no native Windows application (or Mac application for that matter), and their mobile application feels like a lazy reskin of the the YouTube application. This set the tone for the entire experience I was about to endure.
While most content providers such as Tidal, Apple, and Spotify seem to have gone through the trouble to license large libraries of music with many record labels, it seems like YouTube has done the bare minimum required to be called a music service. It’s almost like they were sitting around the board room one day and someone said “hey, people upload music to YouTube, YouTube can make playlists, why not charge for this?” And then proceeded to receive a standing ovation. The quality of the experience is extremely variable. Some songs are from people who just happened to have uploaded the content, some is label-uploaded content via VEVO and others. Then I attempted to import my playlists.
While the experience using Tune My Music was good with Apple, YouTube Music was a nightmare. First since there is no sane user-level way to import, the tool integrates with the YouTube API directly, and promptly hits the 10,000 action limit as it attempts to match and write playlists via the API. My collection still is only about 3/4 imported, having to curate every re-run to get another chunk of my library imported. Then the fun starts - I would say a solid 10% of my library is unavailable on YouTube. Either people haven’t uploaded the music, or Google never bothered to sign detailed agreements with the various labels.
So, after my experience with Apple Music and YouTube Music, I can comfortably say that Spotify is still the reigning champ in my books. Cross-platform applications on Windows and Mac, decent mobile apps, integration with Sonos and most voice assistants. I’ll live with the odd quirks and bugs, at least I can experience them on every platform consistently and cleanly.
]]>Immediately I plugged the disk into my MacBook Pro and opened Disk Utility. After erasing the NTFS partition, I attempted to create a new JHFS+ partition only to be met with the output “Erase process has failed, press done to continue.” Expanding the output displayed the error “Mediakit reports not enough space on device for requested operation.” Confused as I had removed all the existing partitions, I attempted to manually create the new partition as Macintosh Extended (Journaled). No matter the sizing or naming, partition creation would always fail.
After hunting around online, I finally reached a workable solution. I am working on MacOS High Sierra which enabled the newest APFS file system. If you wish to format your external with APFS, you will first need to format it as HFS+, then subsequently migrate it to APFS.
First you need to get the name of the disk you are trying to format. On my MacBook with High Sierra there were 2 existing system disks “disk0” for the recovery files and APFS container volume and “disk1” which is a synthesized set of APFS volumes within the container. As such, the external hard disk appeared as “disk2”, but this may vary on your system depending on what you have mounted.
diskutil list
Once you’ve identified your disk, unmount it.
diskutil unmountDisk force disk2
Once this completes, you will want to overwrite the boot sector for the external.
sudo dd if=/dev/zero of=/dev/disk2 bs=1024 count=1024
Lastly, you will want to partition the disk as JHFS+, including the name for the new volume.
diskutil partitionDisk disk2 GPT JHFS+ "My Passport" 0g
The magic here is removing the boot sector (also called the MBR). The MBR of a disk manages both boot information and the partition table. If that table is unreadable or corrupt, it can render partitions unmanageable. By zeroing out the boot sector of the disk it forces MacOS to create a new GUID partition scheme that it can manage.
Here is the output of the entire formatting operation as run on my system.
pearce at Deans-MacBook-Pro in ~/Projects
$ diskutil list
/dev/disk0 (internal, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *251.0 GB disk0
1: EFI EFI 209.7 MB disk0s1
2: Apple_APFS Container disk1 250.8 GB disk0s2
/dev/disk1 (synthesized):
#: TYPE NAME SIZE IDENTIFIER
0: APFS Container Scheme - +250.8 GB disk1
Physical Store disk0s2
1: APFS Volume Macintosh HD 116.0 GB disk1s1
2: APFS Volume Preboot 19.8 MB disk1s2
3: APFS Volume Recovery 509.8 MB disk1s3
4: APFS Volume VM 2.1 GB disk1s4
/dev/disk2 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *1.0 TB disk2
1: Apple_HFS 1.0 TB disk2s1
pearce at Deans-MacBook-Pro in ~/Projects
$ diskutil unmountDisk force disk2
Forced unmount of all volumes on disk2 was successful
pearce at Deans-MacBook-Pro in ~/Projects
$ sudo dd if=/dev/zero of=/dev/disk2 bs=1024 count=1024
Password:
1024+0 records in
1024+0 records out
1048576 bytes transferred in 0.491402 secs (2133846 bytes/sec)
pearce at Deans-MacBook-Pro in ~/Projects
$ diskutil partitionDisk disk2 GPT JHFS+ "My Passport" 0g
Started partitioning on disk2
Unmounting disk
Creating the partition map
Waiting for partitions to activate
Formatting disk2s2 as Mac OS Extended (Journaled) with name My Passport
Initialized /dev/rdisk2s2 as a 931 GB case-insensitive HFS Plus volume with
a 81920k journal
Mounting disk
Finished partitioning on disk2
/dev/disk2 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *1.0 TB disk2
1: EFI EFI 209.7 MB disk2s1
2: Apple_HFS My Passport 999.8 GB disk2s2
The easiest way to get DBD::Oracle built is against the Instant Client. Download the following 3 packages (or the corresponding 3 for the Client version you desire) to your system, and extract them to a folder on your system. I opted to keep my install local within my /Users directory, but you can opt to install it to /Library/Oracle as well. Simply extract all 3 zips to the same path in your system.
On MacOS Sierra I had to prepare the following steps to ensure a successful build. You may need to adjust the steps based on the location of your Instant Client. The primary issue is the fact that MacOS Sierra does not like the dynamic path linking using @rpath. A fix is to just set your full path into the binary.
# setup the Oracle Instant Client environment
export ORACLE_HOME=/Library/Oracle/instantclient_12_1
export DYLD_LIBRARY_PATH=$ORACLE_HOME
# download and unpack DBD::Oracle
cd /tmp
wget http://search.cpan.org/CPAN/authors/id/P/PY/PYTHIAN/DBD-Oracle-1.74.tar.gz
tar -xzf DBD-Oracle-1.74.tar.gz
cd ./DBD-Oracle-1.74/
# build the module against 12.1
perl Makefile.PL
make
# fix for problem with dynamic linking on MacOS Sierra
install_name_tool -change @rpath/libclntsh.dylib.12.1 \
/Library/Oracle/instant_client_12_1/libclntsh.dylib.12.1 \
./blib/arch/auto/DBD/Oracle/Oracle.bundle
# complete the install
make install
After installing, you may want to do a few things to make your experience easier.
The key to writing high quality software is to ensure that any code you write can be read by another software developer at a later date. Unlike traditional engineering where finalized designs are rarely modified, software by it’s nature is intended to be re-written, improved, extended, and adapted. There is little or no performance cost when formatting your code for readability and providing comments for future developers. Another sure-fire way to improve code quality is to ensure that code reviews are built into your process. Constructive peer feedback allows developers to improve their own coding standards and habits. Implemented properly, it helps strengthen teams by allowing for open and honest conversation without fear of repercussion.
Technically there are several tools and practices that should be used by almost every team to manage a baseline for code quality. The first to enforce a coding standard across your project, and I don’t just mean spaces versus tabs. Things like brace placement (K&R style and the like), naming conventions, and deciding what work should be allowed in constructors. There are many existing style guides such as Google’s Style Guides which can be used as a base for your own style guides. Establishing a standard and a reference in-house also allows some of this work to be automated via critic and linting tools which can automatically format code and point out rules which may have been violated. Beyond code style there are best practices to follow for each language, and for each development style. An example of this would be in object oriented programming, regardless of language or style, concepts like classes having a single concern and subsystem architectures are solid foundations on which to build.
When it comes to personal experience, I’ve learned over my career that privacy and simplicity may require additional thought now, but definitely pays of later. Concepts like ensuring all loops (except main application loops) are properly bounded. There is almost no case for using unbounded while loops or goto statements. Simply by bounding your application entire classes of errors can be eliminated. Second, complex methods should always be broken down into smaller private methods. I don’t stick to the one-page rule-of-thumb, but I do recommend ensuring that your function is only responsible for one piece of logic. Complex methods that change behaviour based on input should likely be different methods and the caller should be responsible for making that distinction. Ensure that any variables and methods that don’t need to be exposed as public are explicitly marked as private or protected. It is easier to grant access to data than it is to revoke it. Verifying and tracing a system is much simpler when classes have a single concern and marking unnecessary variables and methods as private.
Developers must also consider the safety of their code, regardless of where it is running. Frequently drivers, services, and applications are compromised as a way to obtain access to restricted data. Modern systems are frequently written as online systems, exposing a rather large attack surface through things such as API endpoints, micro services, and web services. Frequently attackers will go after these for weak authentication (hard-coded credentials, default administrator accounts for various tools), SQL injection (not properly escaping or binding parameters), and out-of-date packages (outdated copies of WordPress, unpatched OpenSSL libraries). Within code, it’s worth ensuring that all external data is sanitized before being used, and that we don’t have access to more information than expected. For example, when writing an Oracle SQL statement, it’s safer to use binding than direct string injection. When you write your query, you can use placeholder binding like this: select * from foo where bar = ?. When you go to execute, you would specify your placeholder variables. This would ensure that the data is properly encoded by your driver and prevent one of the most common attack vectors. Even more subtle things, like auto-generated API endpoints that unintentionally expose variables because they are marked public instead of private. It is also important to avoid logging sensitive data as logs can be compromised as they often have more relaxed permissions on who can read them. Ensure things like session keys and passwords are not logged by your application.
Another paradigm of safety is ensuring that your classes are properly scoped. When other subsystems and services consume your code, they are encouraged to use anything marked as public or friendly. This can lead to unexpected behaviour if you unintentionally leave variables and methods public. When writing tight units of code, it is important to only mark methods you intend to be consumable as public. These include get/set methods, constructors, and action methods on your class. If you need to enable extensible code, consider offering interfaces which can be implemented, or base classes which can be extended. This ensures that your code retains control over key attributes and structure.
One key thing missed by many developers when coding are edge cases. While the happy path through the application is frequently well covered, unexpected or improperly formatted input is frequently missed. When writing methods, particularly those that interface with other subsystems, it’s worth ensuring that they are appropriately hardened by considering all potential input. This is accomplished via equivalence class partitioning which allows input to be partitioned into classes of equivalent data. Ideally you would then write one test case for each partition to cover the broad spectrum of input. For example, if a method accepts integers we would want to test for values at the minimum and maximum integer value marks. For strings we would want to test null, zero, one, and and maximum length strings to ensure full coverage. For common objects this could likely be generalized to a pattern since the class partitions would be common to the type. Complex input, such as objects more advanced partitions would need to be devised. When edge cases are covered and tested properly, we can be confident that our code is robust and correct.
Exceptions are also frequently mishandled in code. Methods that can throw exceptions aren’t trapped at the appropriate level, or worse they aren’t even correctly propagated upwards. This can lead to unstable or incorrect application behaviour. When writing code it is important to consider how exceptions should be handled. In some cases we may only want to handle a subset of errors. For example a configuration class may try to read a configuration file from disk and get a java.io.FileNotFoundException which we can handle gracefully by using application defaults rather than custom values from a file on disk. In another situation we may find a file on disk and try to parse it only to get an oracle.xml.parser.v2.XMLParseException which we may want to pass up to the GUI to let the user know their custom configuration file is corrupt. When writing exception classes, it’s worth being verbose so that other classes can decide if an exception should be caught or passed upwards. Well structured and handled exceptions can make entire subsystems more robust and facilitate integration with other system components.
This is a fairly controversial topic. Few people like writing detailed documentation despite many developers wishing they had access to cleaner and clearer documentation. Some sources say that the number of lines of documentation should match lines of code. Others say that the code should serve as documentation. My preferred method is to write concise documentation only as needed. If you have followed the previous guidelines for code quality then documentation should act only to clarify complex logic and document API methods within classes and libraries. Examples of this would include covering a basic description for every public method and constructor, and commenting complex regular expressions. When writing, try to stick to unambiguous technical language and use consistent terminology throughout, especially when referring custom components or business-related entities.
]]>At the top level, there are basically three macros that most people concern themselves with: carbohydrates, fat, and protein. There are secondary items that people concern themselves with such as cholesterol and sodium intake, but those can be managed independently of the three primary macros. The typical diet focuses on carbs and cut fat, body builders tend bulk on protein, and keto cuts carbs and focuses on fats. When I say cuts carbs, I mean it cuts them out almost completely. With only 20g of net carbohydrates allowed per day (total carbs minus fiber), the body switches to a process called ketosis where most of the energy that your body uses comes from ketones bodies in the blood instead of blood glucose. Ketogenesis allows the body to generate ketone bodies and cells can generally use ketones as an alternative fuel source. At a metabolic level, cutting carbohydrates forces the body into ketosis which ends up converting fat into energy. Coupled with a caloric deficit, your body will first use up it’s glycogen stores (and associated water weight), then it will begin to use ketones as it’s energy source. With a caloric deficit, you will burn what you eat first, then you will begin to burn stored fat.
On a higher level, eating fat, proteins, and fiber ensures a full feeling at all times, making it easier to maintain a calorie deficit. By focusing on fat supplemented with fiber and protein it allows you to burn fat optimally without loss of muscle mass. By staying in ketosis, your body will opt to burn fat continuously and you will rarely feel hungry. Anecdotally, while in ketosis I slept more consistently, suffered fewer migraines (more on that in a future post), and enjoyed a consistent 5.1 mmol/L blood glucose level. At first I thought that eating all of these natural fats and healthy meats would cost me more money I found that the cost was offset by my reduction in eating out at restaurants.
Despite all the positives of a ketogenic diet, there are some negatives. A negative to ketosis is the lack of food variety. By eliminating all carbohydrates and not just refined sugars it is hard to eat fruits and root vegetables as both are high in natural sugars. Another negative would be the so-called “keto flu” that some individuals experience when their body transitions to using ketones. I have experienced this effect first-hand in the past, lasting about 3 days, during which I suffered from a headache, general weakness, and mild mental fog. This all immediately cleared up for me once I was in ketosis and didn’t return. Another thing to note is even though ketosis will burn fat preferentially, if you aren’t eating a deficit that means your calories will still mostly come from the food you eat, slowing weight loss. Another common feature of the ketogenic diet are plateaus - points where the body is no longer losing weight. While most are temporary, it can be discouraging, and sometimes require a brief change in diet to restart the process (fat-fast, cyclical keto diet, etc).
All said, I have tried numerous fad diets in the past, all had short-term small success, but ultimately lead to failure. I tried the traditional calories in, calories out ideology, but found that not all calories are created equal. There are better calories such as those tied to fiber and micro nutrients, and the body burns macro nutrients differently. I think the 3 months I followed a strict calorie-in, calorie-out diet were the most miserable months of my life. I then tried the ketogenic diet, and while the food restriction at first felt negative to me, I soon found it to be a positive. I could prepare meals days in advance, cook large batches and freeze them. Grocery shopping was simplified as I only stuck to fresh foods on the outer rim of the grocery store. Most important of all, it worked. That is why I am choosing the ketogenic lifestyle again.
]]>Unfortunately monitoring is often the first item to be cut when dealing with budgeting. Some managers think that monitoring doesn’t contribute to the bottom-line of the company, but I would argue that they’re wrong. Monitoring is a vital part of providing a service that is reliable and can be audited. Monitoring tools can assist in preventing security incidents through patch and compliance monitoring. Monitoring tools also assist in narrowing down service performance issues, providing SLA verification, and ensuring that your current infrastructure is meeting your needs. When 3 seconds of waiting decreases customer satisfaction by about 16%, it’s important to ensure your service is reliable and responsive every time. I actively encourage customers to consider monitoring at development time - it’s significantly easier to instrument new configurations and code than to retrofit and find funding when you’re about to go live.
There are many commercial solutions available in the marketplace at many price points. There are SaaS solutions, on-premise solutions, or depending on your infrastructure, integrated solutions. Many tools offer per-CPU or per-Server licensing, but an increasing of SaaS providers are offering different billing metrics including by the hour, by the check, or by the instance. Beyond commercial, there are also many free solutions available for different infrastructure setups. There are pros and cons to each infrastructure layout and each tool, which I will explore in depth in a future article.
Traditionally servers were owned and housed in a data center by an organization. This typically meant a fixed number of assets with total ownership. As such, beyond your applications and services, you frequently had to monitor the health of your physical systems as well as the load and health of your applications. Things like disk state, temperature, configuration, and load were critical to ensuring a high quality of service. Virtualization abstracted away some of these issues, allowing load to be balanced and services to be shifted off troublesome hosts.
Many companies are now choosing cloud solutions for their applications, be it private clouds or public cloud utilities. This changes the nature of monitoring - no longer are you managing fixed assets, you are monitoring a service as you scale with demand. Virtual servers and application instances are spun up and down on demand, and no longer are you concerned with the health of a specific physical machine or even virtual host. This dynamic nature means you need a monitoring solution that can be deployed automatically and monitor the application and service metrics that you care about. Some cloud provides such as Amazon have built out services such as CloudWatch to monitor the status of your service. But these tools tend to scale in cost with your service, and don’t provide fine-grained integration that traditional monitoring tools have enjoyed. Some services like NewRelic have addressed application performance and reliability monitoring, but have high cost and high integration effort.
Depending on the design and nature of your business there are many things to consider when it comes to monitoring. There are also three high-level metrics to consider when designing a monitoring solution for your systems: severity, risk, and preventability. Severity relates to the the impact that an outage would have on your business. Risk relates to how likely an issue is likely to occur. Preventability relates whether an issue can be prevented before it causes an outage. As a result, designing an effective monitoring solution can be extremely complex. When evaluating monitoring tools you need to consider what kind of issues can be captured, and the cost in terms of time and software customization. Some things to consider when evaluating the need and complexity of a monitoring solution:
Traditionally monitoring a system meant installing an agent on a system, and having that agent configured to monitor a number of metrics, potentially with custom scripting. A popular alternative to this is agentless monitoring. A central service (SaaS or otherwise) would connect using established protocols such as SSH on Unix and WMI on Windows, and run queries and system commands against the system. The data would be parsed and managed remotely. The advantage of this approach is you would only need to configure permissions and connectivity to a system. While remote access may be ideal for configuration and deployment, the need to transfer data remotely may violate compliance with regulation such as PCI DSS or HIPAA. Other limitations may be platform dependent - you may not be able to access all application features and information on all OS platforms and security breaches could result in large data breaches.
You may also want to consider the needs of your organization. Application instrumentation and monitoring may be an essential part of your daily operations. Tools like NewRelic can integrate with your application along with off-the-shelf software to provide request and resource-level monitoring. Tools like Nagios an be customized to integrate with your tools as well through custom scripting. Sometimes it is sufficient to monitor the system and high-level components such as the database and core services running on the system.
]]>