diff --git a/blog/marc-andreessen-techno-babble.html b/blog/marc-andreessen-techno-babble.html
index 4d252130..4b6f9ad6 100644
--- a/blog/marc-andreessen-techno-babble.html
+++ b/blog/marc-andreessen-techno-babble.html
@@ -277,7 +277,7 @@ <h1><a name="forward"></a>Forward</h1>
 squeaked out a passing grade in a macroeconomics course.</p>
 
 <p>Were it not for the fact that it’s authored by a famous billionaire, it would
-just be another screed on another blog no one reads.</p>
+just be another screed on another blog no one reads (like this one).</p>
 
 <h1><a name="accelerationism"></a>Accelerationism</h1>
 
diff --git a/fixtures/index.json b/fixtures/index.json
index 3efa39dd..727a2624 100644
--- a/fixtures/index.json
+++ b/fixtures/index.json
@@ -1 +1 @@
-[{"body":" I have extensive experience with:   Agile project management.  Hiring and managing remote teams.  Catalyst, Moose, Template Toolkit, DBIx::Class MVC stacks.  Testing of large-scale applications.  Fixing database integration issues.   I work in many areas, but if it’s Perl-related and your company uses Perl heavily, there’s a good chance that your employees already know me by reputation or have read one of my books. You’ll find me easy-going and diligent. I have a strong background in software testing, having built or worked on many of the most popular testing tools for Perl, including my favorite, Test::Class::Moose , an xUnit framework for large application test suites.  Email me and let’s discuss how I can help you build value for your customers.  A few things I have been hired for:   Building software development teams.  Corporate event keynotes .  Seminars to help companies transition to agile ( sneak preview ).  Rescuing legacy codebases .  Building new software systems in Perl .  Rebuilding or fixing test suites .  In-house A\/B testing (proprietary).  Pharmaceutical ETL systems (proprietary).  Real-time bidding (proprietary).   You might also want to check out:   My LinkedIn profile  My Github account  My CPAN account   Here are some of my articles to help you get a good sense of whether I’m a fit for your needs.  ","title":"Hire Me","url":"\/hireme.html"},{"body":" I’m Curtis “Ovid” Poe. I’ve been building software for decades. Over the past few years, I’ve moved more to the business side, managing projects and teams, training companies how (and when) to be agile, and still writing software and books about software development. I regularly speak at conferences and corporate events across Europe and the US.  I specialize in building database-driven websites and teach developers how to design and use databases. I also specialize heavily in the Perl programming language and joined The Perl Foundation board of directors in 2009.  Useful Links   LinkedIn profile  Github  Twitter  Facebook  All Around The World   Hire Me  Drop me a line and let’s discuss how I can help your company. Naturally, I do a lot of bespoke development on large, database-driven websites using Perl, but there are many other areas I can help you with. I can help get your legacy codebase back on its feet or fix your database issues . I also do agile consulting and can help you understand when to choose Agile . In fact, I’ve been working in Agile (mostly Scrum and XP) for so long and consulted with so many company that I’ve created “WildAgile”, an agile system that maps to how your developers actually work , rather than imposing an artificial, monolithic structure.  Public Speaking  I also speak at private corporate events and technical conferences across Europe and the US and am frequently the keynote speaker. Here’s a video of one of my talks at a Perl conference, but was also rated 5 out of 5 stars at OSCON :     ","title":"Ovid's Programming, Politics, and Prose","url":"\/index.html"},{"body":"","title":"Public Speaking by Ovid","url":"\/publicspeaking.html"},{"body":"","title":"3D Starmap in JavaScript","url":"\/starmap.html"},{"body":"   Tau Station’s universe is dangerous   You’re doing heater runs in Taungoo when a contact you haven’t heard from in cycles asks you to quietly deliver a package to Nouveau Limoges, another station in the Sol System. The money’s good so you head down to the port, hop in Amethyst, and launch. Amethyst’s an older ship and she’s higher maintenance than you’d like, but she keeps flying.  A little over 7 segments later (just over 1.5 hours, old Earth time) you arrive at Nouveau Limoges and that’s when the trouble kicks in. You’re a Consortium citizen, but Nouveau Limoges is a Gaule station and you forgot to renew your visa. Immigration computers notice your status and Amethyst’s nav system is overridden to auto-deport you. Meanwhile, you were supposed to deliver the package in your hold within 8 segments and now it’s starting to change shape. You think you’ve been set up.  Years ago I came up with the idea for Tau Station  , a free-to-play sci-fi narrative adventure you can play in your browser. The intent was to create a universe you can live in, online. As of this writing, we’re still in alpha, but it’s an open alpha and I encourage you to check it out. We’ve developed some fascinating technologies for it and I’ve given a few talks about it. Here&lsquo;s one I gave at FOSDEM  , one of the world’s largest open source software conferences, held annually in Brussels, Belgium, but I like the following one in Amsterdam better:     The Catastrophe  About two centuries ago (Galactic Coordinated Time), civilization was destroyed. No one knows why, though there are many theories.  After breaking free from the confines of Earth, humankind spread throughout the galaxy, setting up colonies on distant planets and hollowing out enormous asteroids for use as space stations. A golden era of post-scarcity was attained. However, humankind was viciously attacked en masse in an event known as The Catastrophe. Planetary defense systems were turned on the planets they were defending. Stations across the galaxy had their air vented, life support systems shut off, and databanks purged. Reactors went critical on ships. Humanity, across the galaxy, was driven to near-extinction in the space of just a few segments. Then the attack abruptly stopped, for reasons unknown.  The Aftermath  In the aftermath, planets were unreachable by those remaining few who were sprinkled out among the stars. For every station with survivors, there were at least a hundred more without. Most human knowledge was destroyed. Books were an anachronism, so the databank purge wiped out most of what humanity needed to continue surviving, as well as the records we had of our past. Those survivors with skills were quickly called upon to spread their knowledge, but their expertise was piecemeal. Without the massive databanks, human technological advances came to a halt. There have even been disputes about the current year, but it’s believed to be the middle 2600s.  Today, no one knows the entirety of how to build a starship, or how to construct one of the massive wormhole jump gates that many star systems have. However, we’ve retained or relearned enough to at least repair them — if we can scavenge parts from dead stations or ships.  This is humanity today. Most of our history has been lost. Much of our existence is a hardscrabble lot, and we few left trying to recover and to simply stay alive. Humankind’s reign has been reduced to a 40 light-year sphere, centered on Sol System. Contact with the planets and most of the other star systems has been cut off. Humans are slowly rebuilding their stations, in hopes that one day, we can regain our former glory.  But who attacked humanity and why? More importantly, are they coming back? There are many theories, but no proof or any solid evidence. Whatever the cause, most of humanity have been raised to be on permanent alert. We’re rebuilding, and if they attackers return, we want to be ready.  The Game  Tau Station is a narrative game. That means it’s a game you read. There are no adolescent boys camping spawn points and screaming improbable things about your mother; It’s a game that lets you take your time, think, and plan.   And because it’s also an MMO, you can interact with other citizens of the Tau Station universe, join a syndicate, explore new star systems, and watch with us as we build the universe.  However, just because it’s text doesn’t mean it’s ugly:    From our tutorial mission   Tau Station is ultimately a murder mystery, with most of the human race being the victim. As the game unfolds, we’ll be revealing more and more of the story, letting players discover for themselves who was really behind The Catastrophe. It’s also science-based (though not hard science fiction). There is no artifical gravity or FTL travel. We use E = mc 2 and the mass of your ships to calculate the amount of anti-matter necessary for travel. We use proper orbital mechanics to calculate the orbits of space stations. And while we have to take shortcuts here and there (it’s a game, after all), we stay faithful where we can.    We still have a lot to do, including:   Syndicate Buildings  “Elite” style trading  Asteroid and ice mining  Ship expansions  Hundreds of new stations  ... and much, much more.   Come join us and see for yourself!  We’re still in alpha, so you can expect changes to come, but we’re pushing hard to create the features we need for launch. I hope you enjoy it; it’s really a labor of love for all of us.  If you want to learn more, read our blog  and checkout out \/r\/taustation on Reddit  .  ","title":"Tau Station","url":"\/tau-station.html"},{"body":"   Preface  The Problem  Background  What is WildAgile?  Benefits of WildAgile  The Workflow  Assumptions Made in Creating WildAgile  The Details  WildAgile Roles  WildAgile Process  WildAgile Remote Teams  Discussion  Extending WildAgile  WildAgile-Compliant  Glossary  Summary     Preface  Some of you may be familiar with Alistair Cockburn  , one of the founders of the modern agile movement. I recently had a brief email discussion with him regarding WildAgile. After reading this article, he replied (and I’m quoting with permission):  I think you should continue with what you’re doing, and get more companies to go along, that will be really great for the world.  WildAgile is an attempt to return agile to its agile roots. So let’s get started.  The Problem  I’ve worked with a variety of successful companies, many of which are Agile in nature, but when discussion of Agile arises, the conversation often goes like this:  Them: We’re agile.  Me: Ah! Do you use Scrum, XP, something else?  Them (embarrassed): Um, er, we just sort of get things done.  And they do! And yet they’re embarrassed by the lack of a name for what they actually do. I often hear something like “we do a modified scrum”, meaning they have sprints and stand ups and that’s about it.  Thus, I developed WildAgile , an Agile methodology that is so agile that it describes what actually happens in the wild instead of what you are supposed to do. It’s designed for very rapid delivery, with Just Enough Process to provide order to the chaos. Though I draw inspiration from Scrum, XP, Kanban and Crystal Clear, there’s a good chance that claiming your team is WildAgile-compliant merely involves reading this and saying “yup, we’re WildAgile-compliant.”  All bolded terms are explained in the description and also in a short glossary near the end. There aren’t many bolded terms because I focus on process, not terminology. The “WildAgile Roles” and “WildAgile Process” sections are all you really need to read to understand WildAgile. Everything else is just providing the rationale. This documentation of WildAgile is released under the CC by 2.5 license . This means that you’re free to do anything you want with this documentation, including sell it, so long as you credit the original authors (just linking to this page is fine). See the license description for details.  Background  When teams transition into an Agile environment, the changes can be intimidating. XP practitioners regularly complain about pair programming (and hence don’t do it). Scrum practitioners often ignore retrospectives. System administrators get upset about a “failed sprint” because they had to stop all work to fix a critical security issue.  And yet work still gets done. Developers finish a task and take a new task. Wash, rinse, repeat. By formalizing this I introduce Just Enough Process to get things done in a structured manner. “Just Enough Process” is an appropriate tagline for management. “Shut Up And Write Some Code” is an appropriate tagline for developers.  What is WildAgile?  WildAgile is appropriate for small teams where the entire code base can be roughly understood by a single programmer. That doesn’t mean that someone points to a given library and the developer knows how it’s coded, or exactly what it does, but the developer should at least have a sense of what it’s for. Because WildAgile is designed to avoid bus-sensitive code , this might be better for small to medium sized codebases, though I know of a huge company, with a codebase exceeding one million lines, which is very profitable and follows a process not quite unlike WildAgile, despite having hundreds of developers. What this means is that as a developer transitions from Apprentice to Veteran (described later), they won’t always know how to solve a problem, but they’ll know where to look to solve that problem.  You may think of WildAgile as a transitional methodology. This means that you may adopt WildAgile before your team transitions to a more formal method (it’s particularly well-suited as a transition to Scrum), but many teams find that WildAgile is all they need.  Benefits of WildAgile   High morale: developers can focus on getting stuff done.  Rapid development: developers can focus on getting stuff done.  Ease of implementation: developers can focus on getting stuff done.   You may notice a theme in this.  That being said, WildAgile is not without its risks. These will be discussed on an ad hoc basis as I discuss its components. However, I do not recommend WildAgile when a potential for real harm occurs. For example, I probably would not use WildAgile to design guidance systems for planes or medical radiology software. I also do not recommend WildAgile for highly complex projects in which the majority of developers cannot understand how various components fit together.  The Workflow  Just to make it clear what I’m focusing on, here’s the development workflow for one company I’ve worked with:    Everyone ignored this and still got work done.   Every task is expected to follow that process. If you have to break a task into new tasks, you’re still expected to follow that process for every subtask. Developers in this company regularly ignore this workflow and still get things done, but management gets frustrated because they can’t see the exact step that every task is at.    WildAgile strives to implement the following workflow:   Things are not this simple in the real world, but I strongly advise clients to understand the Platonic Ideal of Agile and constantly strive towards it.  Assumptions Made in Creating WildAgile  Drawing on work from Alistair Cockburn’s Crystal Clear methodology, I assume that the following three requirements are the most important in developing high-quality products.   Easy and regular team communication  Rapid delivery  Expert customer access   Many experienced individuals feel that if these three requirements are met, a positive-feedback loop is created which allows the team to rapidly build and improve upon products. Any obstacles to these three requirements being met are detrimental to the productivity of any WildAgile-compliant team.  I am also a firm believer in the Pareto Rule: 80% of your benefits will stem from 20% of your actions. WildAgile attempts to implement that 20% of actions that give you majority of benefits of Agile. In fact, I’ve found that for many smaller teams, attempting to implement the other 80% of actions may actually be detrimental as you spend more time fighting process instead of developing software. This is due to a fundamental tension in many Agile methodologies: it’s an attempt to create processes that are designed to minimize process. It’s a delicate balance that is easy to get wrong.  Cockburn’s (pronounced coe-burn , if you’re curious) team requirement actually said “colocation” of the team, requiring team members to be physically near each other. This ignores today’s reality. Savvy companies realize they have access to an entire world of developers if they are willing to have remote workers. Thus, tools such as Slack, Google Hangouts, Skype, Zoom, and other communication channels should be employed to ensure that someone is always available to help. I’ve seen WildAgile work in teams that are also temporally diverse (separated by many time zones), but this does introduce challenges that require extra attention to team communication.  Note: if you don’t have much experience with remote teams, please read my Managing a Remote Team article.  The Details  Yeah, yeah. That’s enough blathering. What’s it all about?  WildAgile Roles  There are only three roles defined in WildAgile.   Developer  Team Lead  Product Owner   Developers , of course, develop the product. They take tasks , finish them, and then take new tasks.  The Team Lead (often shortened to “Lead”) is also a developer. The Lead is also the expert who knows the entire system and can be relied upon to answer questions and provide excellent feedback. Developers decide how they’re going to implement something, but if there is a problem, the Lead has final say. The Lead should defer to the Developer whenever possible and offer constructive advice when appropriate. Team Leads support, not direct.  The Product Owner ( PO ), similar to Scrum, maintains the product backlog . However, there is no “sprint backlog” because there are no sprints. The PO is responsible for knowing the full backlog and being able to answer any business question related to them. It’s preferred if the PO can also answer technical questions as well, but I realize that it’s often hard to find someone who is both a business expert and a technical expert. Thus, the PO focuses on business excellence and the Team Lead focuses on technical excellence.  Note: if you’re new at this and you’re overwhelmed by prioritizing the product backlog, read my article “Project Management in Three Numbers” . It makes this much easier.  Additionally, the PO serves as the proxy for the customer. They are the “expert customer” that you should be able to rely on at all times to answer questions about the product. The PO is encouraged to practice an embedded anthropology approach to knowing their customers. This means that they should use the product, but also work directly with real customers (both new and existing) to understand customer needs and desires.  As an aside, the customer should always be referred to as “the customer” and never as “the user.” In English, the word “user” can have a negative connotation and in our experience it is often used in a condescending manner. Customers are people too.  WildAgile Process  Every day, a 15-minute time-boxed stand up is held. It’s recommended to be at the end of the day to ensure that what you did is fresh in your mind, but this is not a requirement. The stand up begins with the Summary . Before the stand up, the product owner should briefly discuss where you’ve been and where you are going. Any interesting future issues for the team should be mentioned at this time. The benefit of the Summary is that it helps remind developers of the context of their work and better understand upcoming priorities. This makes it easier to choose appropriate tasks.  After the Summary, each team member should explain:   What they did today  What they’ll do tomorrow  Any blockers they have   On a daily basis, every team member is allowed, even encouraged, to improve the code base. Add a Makefile target to rebuild the project. Factor out common code. Move test data into fixtures. The developer is expected to mention this in the stand up and it is the responsibility of the Team Lead and the PO to rein in developers who spend too much time “improving” instead of “developing” (and vice versa!).  Tasks are “things a developer can do.” Every task is done in a separate branch (when appropriate) and when it is done and the tests pass, it is merged back into the “master”, “trunk”, or “integration” branch, or whatever you call it. Tasks are expected to include tests, when feasible, and if merging the code breaks any test, the developer responsible for merging the code should be the person responsible for fixing the tests (again, when feasible). Details about how to manage source control are left to the team.  Tasks should be as short as possible. Any task expected to take over a certain amount of time (a week is a good limit) should, if possible, be broken down into smaller tasks, each of which can add value even if the overall task is cancelled.  When a task is finished, the developer independently chooses their next task, keeping in mind the context provided by the last Summary. They should prioritize tasks assigned to them, high value tasks, and quick tasks. It is the responsibility of the Team Lead and PO to guide developers to suitable tasks if the team member is regularly making poor choices on their next task.  If enough open tasks are open, it is the optional responsibility of the PO to designate Priority tasks. If a new task is to be chosen, they should be chosen from Priority tasks. If a team member is unconvinced of their ability to accomplish any Priority tasks, they should speak to the Team Lead for guidance.  Obviously, if a critical issue arises, any task can be stopped and the Team Lead can assign someone to tackle the issue. There is no formal structure to this because WildAgile is designed to be as lightweight as possible.  New team members ( Apprentices ) are initially encouraged to focus on a single area to become very comfortable with the business rules. After the Team Lead and PO agree that the developer has become comfortable in their area, they become Veterans . Veterans are strongly encouraged to regularly choose tasks in areas of the code base they have less experience with. If Veterans do not regularly work on different areas of the code, they may find themselves focusing on a small subset of the code base. Not only does this limit the Veteran’s ability to contribute to the code base, it encourages other developers to not work on areas of the code that it’s assumed the Veteran will work on. It is the responsibility of the Team Lead and the PO to ensure that Veterans are regularly choosing tasks outside of their area of expertise in order to spread knowledge of the system.  Team Leads and POs do not promote Apprentices to Veterans until the Apprentice:   Appears to understand the business (not the code base!)  Is writing solid code  Is writing appropriate tests   Releases are not described in the WildAgile process, but it’s suitable for both continuous deployment and regular releases (as one might expect from a sprint or iteration).  Note that in our experience, Apprentices get promoted to Veterans fairly quickly and in established teams, you generally only have Veterans, thus reducing the distinction between the two.  WildAgile Remote Teams  Many companies are discovering the competitive advantage of having remote workers:   You have access to a much wider pool of developers.  You don’t need to pay for office space.  Developer salaries are easier to control.   That last point is what is driving much of this and is causing some industry disruption. When you realize that your $150K a year developer can be had for $30K, that opens up possibilities you may not have had before, particularly when you find that expert you couldn’t find locally.  However, companies often feel uncomfortable with remote workers because they’re less used to this. WildAgile works well with remote workers, but a key change needs to be made: stand ups.  For colocated teams, time-boxing a stand up to 15 minutes works well because if there are issues later, a developer can simply walk over to someone and ask. That doesn’t work with remote workers. Instead, you simply extend the stand up time. I recommend 30 to 45 minutes. Yes, this means you’ll be spending more time in the stand up, but you can communicate more about what you’re doing, and how you are doing it. This helps to catch problems early on and minimizes the need for developers to distract other developers while they’re working.  Discussion  The idea behind WildAgile is not to formalize a new methodology, but to put a name on a practice that is already moderately well-known but followed in an almost embarrassed manner. It’s a way of following a set of reasonable development guidelines that can eventually move on to a more appropriate Agile development process. Think of it as “Agile light”, if you will. By putting a name to it, you can say “we are WildAgile-compliant” instead of “eh, we just get things done.”  In short, WildAgile eliminates the silo problem of cowboy coding, but allows the extremely rapid development of products. It encourages creativity while having Just Enough Process to keep development on target.  By separating developers into Apprentices and Veterans, I create the psychological incentive to explore more of the code base. Ambitious developers quickly want to shed the Apprentice label and fully participate in the rest of the team’s work. However, by starting out by becoming an expert in one part of the system, the Apprentice can quickly gain confidence in their ability to participate and make a meaningful impact on the system.  Hiring for a WildAgile process is important. You want developers who are comfortable in a fluid environment, but can also appreciate business concerns. There is a tremendous amount of freedom in WildAgile and it’s very easy for someone to get carried away and try to “perfect” (*cough*) everything at the expense of adding value to the company. Thus, both the Team Lead and the PO are required to keep things in check.  Also, WildAgile benefits from skilled, conscientious developers. It’s very easy to spend too much time “perfecting” your code, but it’s also easy to spend too much time pushing out new features and ignoring technical debt. This is a hard balance to maintain. WildAgile does not address this directly because every project is different and every company is different. However, as a guideline WildAgile suggests that the PO advocate for features and the Team Lead advocate for technical debt management, but each should be able to understand and even agree with the other’s point of view. If you can’t balance these two needs, you’re likely to have long-term problems.  In practice, I find this development model to be very successful for small projects. The pace of development is very rapid and I don’t find that code quality is significantly worse than for other methodologies. The most significant obstacle that I find is appropriately managing the backlog. Typically backlog items are merely added to a ticketing system, such as Jira, RT, Redmine or Trac. This does not scale well unless the PO is competent at managing the tickets and creating appropriate Priority tasks.  Extending WildAgile  Do you like retrospectives? Do them. Do you need milestones? Great! Do you like pair programming? That’s fine. WildAgile is designed to describe the core of what successful development teams are already doing. Items outside of this core are deliberately unspecified to allow you to customize WildAgile to your heart’s content. A guiding principle of WildAgile should be “whatever works for you.”  You are encouraged, nay, required to customize WildAgile to meet the needs of your team.  WildAgile-Compliant  Many teams are already WildAgile-compliant without even knowing it. If you follow Scrum, you’re probably WildAgile-compliant. If you follow XP, you’re probably WildAgile-compliant. If you do ad hoc development, you may already be WildAgile-compliant. Don’t stress too much about it. I don’t envision WildAgile-compliance courses. I don’t envision WildAgile-certifications (though if you pay me enough money, I’m happy to give a one-hour presentation and print up WildAgile-certifications for you).  Glossary   Apprentice A new developer becoming an expert on part of the system.    Backlog Tasks which have not yet been done.  Critical Issue Something which must be done now .  Just Enough Process Only enough process to provide structure.  Priority Tasks which should be taken before other tasks.  Product Owner (PO) The proxy for the customer. Maintains backlog and assigns Priorities.    Stand up An end-of-day “what I did, will do, and blockers” explanation for every developer.  Summary The PO’s business-focused summary of the project status, given immediately prior to the stand up.  Task Something a developer can do.  Team Lead The code base expert. Should also understand business concerns.  Team All developers, including the Team Lead.  Veteran A developer who is allowed to work on all parts of the system.  WildAgile “Real World” Agile.   There is very little terminology and most of this fits what you already know.  Summary  Shut up and write some code.  Seriously, there is nothing radical about the above and it mirrors what people are actually doing rather than forcing them to adopt a new methodology all at once. You might view it as a gentle transition to more formal Agile methodologies. Or, if it’s working fine for you, it might remain your Agile methodology. Its very loose structure presents some risks, but a key component of managing risk is being aware of it.  ","title":"WildAgile","url":"\/wildagile.html"},{"body":"    Why am I Writing This?  1984  Cheating on Tests  Cheating on Homework  The Break-In  Interlude: The Tax Office  Getting Caught  Aftermath  Tax Office?     Why am I Writing This?  This is here because when I reread the eulogy for my father , I realize that no one knew much about his life. This isn’t here so much as to share my life with the world, but to start sharing snippets of it with family members after I pass. There might be some bad things here, but I know how painful it was to try and reconstruct my father’s life, so I don’t want my family to go through the same thing.  It might prove to be mistake. I had a rough life growing up and some stories I might share do not reflect well on me.  1984  This is the story of how I got caught breaking and entering in my high school in Texas. Names of guilty parties have been changed because most of them are still alive and it’s not fair to taint them with high school stupidity. Also, this happened 30+ years ago, so it’s possible my memory is shaky on some timelines.   It will come as no surprise to most readers that teenagers are stupid. There’s interesting research explaining why  , but it boils down to having brains that aren’t fully developed  until their mid-20s. I think mine didn’t develop to my mid-30s, but that’s a story for another day.  I had been referred to in high school as a “goody two-shoes.” That’s because, well, I had a reputation for being what we in France refer to as « carré ». The word translates to “square”, but in this context, it tends to mean “someone who is correct and doesn’t break the rules.” (It’s possibly less pejorative than “straight-laced.“)  In short, I was a rather unpopular nerd. I didn’t smoke, do drugs, swear, and when I was once asked if I masturbated, I instantly replied “of course not!”    I was as pure as the driven snow, except for my militant atheism thing. But otherwise, I simply didn’t break the rules.  So it was rather a shock when students found out that I, too, was a rebel.    Yeah, that’s me looking like an idiot on the upper left.   Cheating on Tests  My first high school, Keystone  , was a private high school for the academically gifted. My grandmother, who had been paying for the school, pulled me out after two years because, frankly, I didn’t care about my grades. I didn’t have a great childhood, so I put only enough effort into school to not fail. But my geometry teacher took pity on me and let me play around in the computer room, a room reserved for grades 11 and 12.  I taught myself BASIC and when I was returned to Marion High School  in Marion, Texas, I found myself in a much easier school and suddenly, I was getting A’s. It helped that I was hanging out in the computer room and was able to get copies of the teacher’s tests. Another student we’ll call “Alex” was a teacher’s aid and he’d swap the tests he would type up for my class with the tests I would type up for his class.  But one day, he said “no” to the mid-term. Seems the teachers were getting wise and put pressure on him, even though he didn’t admit it. But he also told me that it didn’t matter because his disk had crashed (in the 80s, we’d use floppy disks that easily got corrupted) and he couldn’t print out our mid-term even if he wanted to.  No matter. We couldn’t keep our floppy disks, so we had to return them to the desk when we left the room. I had already acquired the master (root) password for our TRS-80 computers and I was able to retrieve his disk and do a raw disk dump straight to the dot-matrix printer. I had most of the test, though some of it was corrupted (the essay questions).    An old, TRS-80 computer with dual floppy drives. Source    I gathered several students and we hit a truck stop restaurant to study for the mid-term. And then one of them said, “I really like Mrs. Womack. I can’t cheat on her test.” And one by one, all of the students agreed we couldn’t cheat. I did, too, but it was because I was lazy.  So we had a nice time at the truck stop and the next day, I didn’t get an A on my test.  I later found myself sitting at a computer (surprise!) after physics class when Mrs. Womack burst in, furious.  “I thought I could trust you, Curt!” (Everyone called me “Curt” instead of “Curtis” back then). “I just found out you had a copy of the mid-term.”  I looked at her and calmly replied, “if I had a copy of the mid-term, would I destroy my GPA like that?”  Her eyes widened and she stopped. And then she apologized. Of course, it was stupid that I would do such a thing. I never heard another word about it.  Cheating on Homework  Of course, then, as now, I loved computers. One of our assignments was doing ten key by touch  . To say that it was boring would be an understatement. We’d start with seeing a random series of ones and zeroes and have to type them in as quickly as possible. When we had a good enough score and speed, we’d move to zeros, ones, and twos. And so on.  I wasn’t going to do this. Since I had the master password, I could read and analyze my result output. After doing that, I wrote a program that would simulate my error rate and typing speed and simply write out my results so I could go on to playing with the computer instead of doing boring work .  Today, of course, I’m horrified by that. I should have analyzed the standard distribution of my typing speed and error rates and ... I mean I would never have done such a dastardly thing. But hey, I was a teenager.  So while the other students in the class were slogging through their ten-key exercises, I would run my covert program and go back to playing.  But ... these were TRS-80 computers with dual floppy drives. You put the master floppy in the top and your floppy in the bottom. If you got it wrong, your scores wouldn’t save.  Mine would because I had the master password. And if I had the disks reversed, I would overwrite the master floppy. And I had gotten the disks reversed. “Oh no!”  Mrs. Bode (pronounce boh-dee) looked up and asked, “what did you do, Curt?”  I had played around enough that I had “crashed” quite a few floppies and was told that I would not be allowed back in the computer room except for proper classes if I did it again. My first thought, after realizing I would be caught cheating, was that I wasn’t going to be allowed back into the computer room. Disaster!  “Nothing, Mrs. Bode. I crashed one of my own disks, not yours.”  Naturally, I had overwritten the master disk and while I could cover up my crime, I couldn’t hide the fact that I had crashed another school disk. The Master Disk. I was going to be barred from the only thing in school that I truly loved.  So I approached “Alex,” my erstwhile partner in cheating and explained the problem ... and my plan. He listened and said he wouldn’t help.  I was stuck. So I approached another friend, “Bob.” Bob was a player in the AD&amp;D (Advanced Dungeons and Dragons) campaign that I was running and he was instantly eager to help with an adventure!  Except we needed transportation. Enter “Charlie.” Charlie was also a player and agreed to help, but with a caveat. For (identifying) reasons I won’t discuss, he was much more at risk if he got caught, so Bob and I agreed to keep him out of it.  The Break-In  That night, Charlie picked me up at my house and we went and got Bob. Charlie was driving a pickup truck with a ladder in the back and as we got to the school, Bob and I took the ladder and Charlie drove off. He agreed to come back in an hour.  Bob took the ladder and placed it on the side of the school and he and I climbed to the roof. At the time (I don’t know about today), there was a large, open-air space in the school—see the photo above—and if you dropped down into that from the roof, you were inside the school.  The locks were shabby and anyone with a pocketknife, something most farm boys carried at that time, could jimmy a lock with ease. So Bob shimmied down a column into the school and immediately gestured for me to stay where I was. Seems that in the building next to us, but clearly visible down the hallway, was the band practicing. Bob grabbed a desk that had been left lying around, flipped it on its side and crouched behind it while he worked on the lock.  When he got it open, he motioned for me to come down. I shimmied down a column and when he motioned the coast was clear, I crept into the classroom after him.  Of course, we were wearing gloves as we rounded up every floppy disk we could find. But I knew that Mrs. Bode might remember the incident with me, and I was paranoid. So Bob wrote on the board “Mrs. Bode is a bitch.” Bob, like Charlie and myself, was such a “good” students that there was no way anyone would suspect us of writing that. With no witnesses, no fingerprints, and a message to throw them off the scent, we committed the perfect crime.  As we left, we hid in the bushes with the disks until Charlie showed up again. We clambered into the truck and drove off. Bob kept saying over and over again about how exciting that was, just like a D&amp;D adventure. Charlie and I shared a look. We knew then that we were screwed.  “Bob, you can’t tell anyone, right?”  “No, I won’t tell anyone.”  Interlude: The Tax Office  Let’s take a little diversion. At this time, I had heard plenty of rumors that our school had been doing dodgy things to get more tax income. Schools in the US are generally funded through income taxes, sales taxes, and property taxes. If you live in a poor area of the US, your school is miserable because it’s underfunded.  Rumor had it that our school had figured out a way to get more property taxes. In fact, our school even had its own “tax office.” It was a room in the school that was filled with computers, that students didn’t enter, and allegedly helped with collecting taxes for the entire Marion School District. How much of this is true, I have no idea, but hey, rumors! They’re chum for the adolescent sharks.  The tax office door, unlike the other doors in the school, had a steel plate blocking off the latch, so you couldn’t easily jimmy it open with a knife. Not a problem. I had a acquired a key. And I knew where they had written down their computer passwords. I was going to break in and bust the whole corrupt scheme wide open. Like Bob, I was hungry for adventure.  I never did break into it, but it became relevant later, by a very weird coincidence. There were a couple of weird coincidences that day and the reason we got off so lightly may have been because the school had other matters to worry about in what had to be a complete day of hell for the school.  Getting Caught  The next day at school, everyone was talking about the break-in and it was scary. Bob, Charlie, and I gathered around and I reiterated my position. There were no witnesses, we had worn gloves, and no one had any reason to suspect us. No one is to say a word to anyone, right? Bob and Charlie both agreed.  At the beginning of physics class, Bob was called to the office. Charlie and I looked at each other and knew we were screwed.  Later, Bob came back. He was in tears.  A word about Bob. Bob was a great guy. He was a jock, but a tall, good-looking, extremely intelligent jock and an all-around nice guy. He didn’t deserve this, but I had gotten him into it. Bob had previously caught by his father of committing the heinous crime of playing AD&amp;D with me and this was going to be the last straw. Bob’s father was going to mad and Bob was terrified.  A quick, in-class whispered consultation revealed what happened. Bob had told “Dan” who promised he wouldn’t tell anyone. Dan had told “Evan” and, as near as we can figure out, Evan told everyone.  Had Bob given up Charlie? No, he kept his name out of it. But yeah, the school knew about me. Good, I wouldn’t mention Charlie either. It was my fault and I didn’t want him to get in trouble too. He had too much at stake (I’m deliberately not saying why because it’s identifying).  So when physics class ended, I was rather curious as to why I wasn’t called to the office. Yeah, they knew what I had done, but they hadn’t called me down to the office. What the hell? Something odd was going on. So I skipped my next class and went to the office. When the secretary saw me I simply said, “I think the principal wants to see me?”  She said, “yes,” and pointed to his door.  The principal wasn’t there, but our history teacher was there. He was writing a letter that he quickly covered up before hastening out of the room. Seems it was his resignation letter. He was caught in a compromising situation that, at that time, wasn’t a crime, but was serious enough to demand his resignation (not, I’m not going to say what happened). The principal was, of course, a touch distracted by this.  The principal arrived, along with the school district superintendent. They sat down and began to describe what they knew (a tactical mistake, because it told me how to play my story). Two students had broken into the school, the police were called, and there were witnesses and fingerprints. What did I want to say before the police came?  Well, that was interesting. Two students, not three, so Bob had kept his word. This was in the middle of nowhere so, unless we were spotted by someone at band practice—something I felt was unlikely—they lied about witnesses. And I knew they didn’t have fingerprints.  So I told them the truth, the whole truth, and nothing but the truth. Except that my truth left out Charlie. Out of this entire mess, Charlie was never caught.  “You’re lying.”  “No, I’m not!”  “You broke into the tax office!”  The hell? At this point, my mind is reeling. Yes, I had planned to break into the tax office, but I hadn’t done it yet. Had someone else broken in the same night? Had the school found out about my plan to try to expose tax fraud? Had the person who gave me the key told them how I got the key? What the hell?  “I didn’t break into the tax office!”  “We have fingerprints. You’re going to be arrested.”  I continued to protest my innocence and they continued to insist that I did it until I got mad and yelled, “I didn’t do anything!”  “You stole the floppy disks.”  “Oh, yeah.”  Aftermath  Charlie was never implicated. Bob and I received in-school suspension, were barred from the class trip, and I apologized to Mrs. Bode, explaining that we only wrote that she was a bitch because we knew no one would believe we thought that. She was surprisingly gracious about it. Apparently, she understood teenagers better than teenagers do.  And as our final punishment, Bob and I were forced to complete the ten-key by touch homework for every other student in the class. All of our free time was spent in the computer room doing that. I got rather good at ten-key by touch and can still do it to this day.  Of course, I worked out the math. I pointed out to Mrs. Bode that it was literally impossible for us to finish all the students ten-key assignment by the end of the year. But she was unmoved. We had to do it.  And then she was gone for a week. We had a substitute teacher and I, not having properly learned my lesson, took the program that I wrote and hastily completed most student’s work by the end of the week. When Mrs. Bode returned, Bob and I only had a few more day’s worth of work to do.  I’m pretty sure that Mrs. Bode knew that we had cheated, again. But she never said a word.  Tax Office?  So that’s the end of the story, but you might be wondering what the hell was going on with the tax office break in.  As I found out later, there wasn’t one. That’s what the school wanted to have me arrested for. As it turns out, the school’s computers couldn’t connect to their tax system and the school thought I had something to do with that. I had been planning it, but they didn’t know that. Instead, when I was breaking into the school, AT&amp;T was breaking up  . Due to an antitrust action, AT&amp;T was forced to break up into seven smaller phone companies and did so when I was breaking into the school . The next day, the school’s computers appeared to be “broken” and given that I was the school “computer kid” who kept messing with computers, it was assumed that I was responsible.  ","title":"How I Got Caught in High School","url":"\/blog\/how-i-got-caught-in-high-school.html"},{"body":"    Forward  Accelerationism  Techno-lust  Free Markets  Capitalism  Market Discipline  Luddites     Forward  In the vein of Ted Kazinsky, Karl Marx, and others before him, Marc Andreessen has proudly penned a manifesto, The Techno-Optimist Manifesto  . Or more properly, “a rant.”  I chose the word “rant” with care. He’s ranting against something, but it’s a terrible essay riddled with citations from racist writers (whom he admires), straw men, false dilemmas, and reads like a high school sophomore barely squeaked out a passing grade in a macroeconomics course.  Were it not for the fact that it’s authored by a famous billionaire, it would just be another screed on another blog no one reads.  Accelerationism  The central thesis is the call for “accelerationism”:   We believe in accelerationism – the conscious and deliberate propulsion of technological development – to ensure the fulfillment of the Law of Accelerating Returns. To ensure the techno-capital upward spiral continues forever.   It’s possible that Andreessen isn’t aware that fascists and white supremacists love accelerationism  , but given all of the authors he cites, including admiration for “the father of accelerationsim,” Nick Land, I strongly doubt this. Nick Land, in particular, is often accused of advocating for “scientific racism” and eugenics  .  Andreessen also calls for free markets and for people to stop hating technology. The following assumes that he’s arguing in good faith and not deliberately using deceptive arguments. Also, libertarians are, demographically, overwhelmingly white males from rich countries and have long harbored white supremacists  , but there’s too much to unpack there, so that is (unfortunately) being left our of this response.  Techno-lust  Let’s get the easy one out of the way.  Andreessen uses the word “technology” 48 times in his rant. He opens with this:   We are told that technology takes our jobs, reduces our wages, increases inequality, threatens our health, ruins the environment, degrades our society, corrupts our children, impairs our humanity, threatens our future, and is ever on the verge of ruining everything.   This encompasses two central themes in his work: the role of technology in society and Andreessen’s inability to understand what people are complaining about.  Let’s say I visit Andreessen’s house and I see a nail gun lying around. It’s a bit of technology which either makes carpenter’s lives simpler or takes work away from them, depending one’s point of view. I wouldn’t care if Andreessen has a nail gun.  If, however, Andreessen was using that nail gun to kill puppies, I would definitely care and Andreessen would apparently assume I have a problem with nail guns.  Dear Marc: it’s not the thing we object to, it’s the abuse. As we read this on a smart phone or a computer, as we ride our bikes to work, as MRIs diagnose diseases and save our lives, few of us are objecting to those things.  Yes, there are certain forms of technology we object to because they’re inherently harmful—internal combustion engines (ICE), for example—but until we have a suitable replacement, no one’s calling for outlawing ICE tomorrow. Even when Europe banned the sale of new ICE after 2035  ,   the time frame was an explicit acknowledgement of the benefits of this technology, while also pointing out the obvious dangers of it.  It’s not technology people object to. Andreessen’s argument is a straw man. He extols the virtues of technology with drooling ecstasy, ignoring that virtue stems from humanity, not tools.  Free Markets  Andreessen writes, “We believe free markets are the most effective way to organize a technological economy.” Two paragraphs later he then rails against central planning.   We believe Hayek’s Knowledge Problem overwhelms any centralized economic system. All actual information is on the edges, in the hands of the people closest to the buyer. The center, abstracted away from both the buyer and the seller, knows nothing. Centralized planning is doomed to fail, the system of production and consumption is too complex. Decentralization harnesses complexity for the benefit of everyone; centralization will starve you to death.   Hayek’s “Knowledge Problem” refers to the famous article The Use of Knowledge in Society  (pdf), by Friedrich Hayek. In this writing, he skewers the notion that central planning can lead to effective economic outcomes. It’s a good, important work and should be read by anyone who wishes a better understanding of the limits of central planning.  Andreessen sets up the great debate, writing “centralized planning is doomed to fail, the system of production and consumption is too complex.” He attacks communism several times, to make it clear that it’s the communist bogeyman he’s warning us about. He also positively cites libertarian works to let us know that laissez-faire is our Savior and Lord.  There are two little problems with him setting up the epic battle of free markets versus central planning. The most obvious is that no one is arguing for central planning. Marx’s ideas, like Marx himself, are dead. There are few economists who take him seriously. Even the Chinese Communist Party has shifted to a “socialist market economy”  , recognizing that central planning doesn’t work. Communism and central planning are safely in the hands of barstool philosophers and pose no threat.  The other problem is that he’s setting up a false dichotomy  . In a tremendously complex world with radically different value systems across its face and mounting climate and ecological chaos, reducing one of the core debates to naïve dualism isn’t enlightening anyone.  However, his writing isn’t always bad.   We believe there is no conflict between capitalist profits and a social welfare system that protects the vulnerable. In fact, they are aligned – the production of markets creates the economic wealth that pays for everything else we want as a society.   I’ve met many a libertarian who argues strongly against welfare on the grounds that people should tend their own estates and government handouts take away our incentive to succeed. Tell that to a single mom working at McDonalds to support two children.   So Kudos to Andreessen for suggesting that social welfare systems are OK.  Curiously, Andreessen later writes, “We believe a Universal Basic Income would turn people into zoo animals to be farmed by the state.” This was stuck in the middle of the manifesto like a porcupine in a petting zoo. I read the text around that several times and I still have no idea what it means, but like most of the rant, it was a bizarre assertion without justification.  Capitalism  While ranting about communism, telling us that only free market capitalism can save us, he blithely ignores the central problem with capitalism today. Yes, he is correct about the power of markets. Prior to capitalism, the world was largely guided by mercantilism  . Capitalism supplanted mercantilism by proving that economics wasn’t a zero-sum game. Eventually, we should hope for a post-capitalist society that addresses the fatal flaw of capitalism: it has no mechanism for handling externalities  .    I don’t know what a successful post-capitalist economic system would look like any more than a mercantilist would have predicted capitalism. Two of the most significant externalities we face today are climate change and ecosystem collapse, but we clearly see that laissez-faire capitalism can’t deal with it effectively. This isn’t just some lefty idea. The US Department of Defense considers climate change a national security priority  The US military isn’t traditionally associated with radicals or lefties.  So Andreessen, by pulling the typical libertarian trick of ignoring the central problem with capitalism, successfully guts his own screed.  Market Discipline  This part is just bizarre. Andreessen writes:   We believe in market discipline. The market naturally disciplines – the seller either learns and changes when the buyer fails to show, or exits the market. When market discipline is absent, there is no limit to how crazy things can get. The motto of every monopoly and cartel, every centralized institution not subject to market discipline: “We don’t care, because we don’t have to.” Markets prevent monopolies and cartels.   I confess that I laughed myself silly at that last sentence.  OK, before we unpack this, we need to understand what “market discipline” is. Per Investopedia  :   Market discipline is the onus on banks, financial institutions, sovereigns, and other major players in the financial industry to conduct business while considering the risks to their stakeholders. Market discipline is a market-based promotion of the transparency and disclosure of the risks associated with a business or entity. It works in concert with regulatory systems to increase the safety and soundness of the market.   There are a couple of interesting things to note about this. First, it talks about risks to stakeholders , nobody else. Risks to customers can become risks to stakeholders, but only if the customers realize that risk. Risks to non-customers, such as from pollution? Who cares? Not their problem. And while Andreessen approvingly quotes Adam Smith about the nature of “self-interest,” there’s another quote of Smith’s which Andreessen didn’t bother to include  (pdf):   People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices.   That, by the way, is from a law review paper entitled “Adam Smith on the Inevitability of Price Fixing.” The quote further reads:   It is impossible indeed to prevent such meetings, by any law which either could be executed, or would be consistent with liberty and justice. But though the law cannot hinder people of the same trade from sometimes assembling together, it ought to do nothing to facilitate such assemblies; much less to render them necessary.   There is no market mechanism to protect against this. The only recourse is in the second interesting thing in Investopedia’s definition of market discipline: regulatory systems. The market’s internal mechanisms for protecting the public have a core problem: businesses are comprised of people and sometimes people think they can get away with stuff and, sometimes, they can. If you decide that Coke, Pepsi, and other products are bad, you can easily switch to water. But what if there is no substitute? Think “insulin.”  And if the market is so great at preventing cartels, why is the US Department of Justice on a tear, prosecuting so many of them  ?  This naïvete about “market discpline” is a symptom of libertarian groupthink.  It’s also worth reading The Market for Lemons  (pdf), an economic paper that won the author a Nobel Prize. It describes how markets degrade when the seller has more information about their product than the buyer. This is called “information asymmetry” and there is, again, nothing in the market mechanism which can guard against this. Even in our “information age,” the buyer often has little way of knowing if the product they’re buying has defects.  Or you can read about the four-square  , an unethical, but perfectly legal technique into pressuring people to pay too much money for a car. Again, the market can’t prevent this. In fact, it incentivizes this unethical behavior to the detriment of the consumer.  Luddites  In addition to railing against the non-existent communist threat, he implies that those who object to our current situation are Luddites:   This upward [techno-capital economic] spiral has been running for hundreds of years, despite continuous howling from Communists and Luddites.   Luddites? In the early 19 th century, the Luddites were  :   British weavers and textile workers who objected to the introduction of mechanised looms and knitting frames. As highly trained artisans, the new machinery posed a threat to their livelihood and after receiving no support from government, they took matters into their own hands.   They were being replaced by low-skilled workers and many of the skilled workers found themselves out of work and starving.  At first blush, this does seem to resemble people like me, worried that my skilled labor is going to be replaced by AI , but this is a faulty comparison.  Eventually, the financial boom created by the Industrial Revolution translated into new jobs and an increased standard of living  . The interim was rough for workers because the government was only acting in the interest of the wealthy and it took decades before those workers found skilled work that the machines could not do.  With how powerful the new AI is in its infancy , where will be the skilled work the AI can’t do? People are already using ChatGPT to help them build entire apps  and plenty of people are losing their jobs to ChatGPT  . As this new technology improves, more jobs will be lost. Unlike the Luddites of old, today’s Luddites are facing a future where there will be few jobs this technology cannot do. When we fully combine it with robotics to replace manual labor, the future looks grim.  What are we to do when we’re losing our homes and can’t afford to eat? Presumably, Andreessen’s answer will be something along the lines of “let them eat cake.”  More likely, since he’s a billionaire, he adheres to the unofficial libertarian motto: “screw you; I got mine.”   Note: because there’s simply too much to cover here, I’ve included an annotated copy of his manifesto here  . Comments are open, so feel free to leave them there or here.  ","title":"Marc Andreessen: Techno Babble","url":"\/blog\/marc-andreessen-techno-babble.html"},{"body":"  Telling Stories    Me and my daughter.   One day, when my daughter was three, she asked me to stop reading stories to her. She wanted me to tell them. So I did. I created a world of faeries and humans and whimsical magic where my daughter would routinely defeat the “evil math faery” by solving a math problem that would break the math faerie’s spell.  She started by counting on her fingers, laboriously working out what one plus two meant. Over the years as the stories evolved, she’s moved on to solving algebra problems, math in different bases ( it’s easier to explain this to younger children ), discovering that the roots of some numbers have have two solutions, and so on. Unsurprisingly, she now excels at math in school.  But she’s getting older. She’s reading books beyond her years, devouring them like a famished dog in a butcher’s dumpster. So I thought it was time to start telling a different story. My intent was to to write a middle-grade novel  , a short work appropriate for those between the ages of eight and twelve. But there were some obstacles.   She often reads beyond her years  It’s taking time and she’s growing older  I can’t seem to help but write “older” material   Thus, my middle-grade novel has morphed into a young adult novel and I’m finding it much more pleasurable to write.  The Reluctant Prince  A classic “many worlds” novel, “The Reluctant Prince” centers around a young boy with autism named “Billy.” When his grandfather is killed and his mother is kidnapped, Billy learns that not only was the man not his grandfather and his mother was keeping secrets, but Billy might be the actual target. A long-lost twin sister, Sia, arrives from Amethyst, the central world. Billy—high school nerd—learns he is probably the heir to the throne of Amethyst...if he can stay alive long enough.  Billy and Sia escape Earth through a slip (a portal to another world) to the underground world of Thonios. Soldiers are already hunting them in this world. There, they’re captured by Hendrix and his nephew, Harris, both soldiers in the Thonian army. The young Harris, who obsesses about his “duty,” is very gung-ho about turning in Billy and Sia as spies, but Hendrix knows that turning the heir-apparent over to a Great House that also claims the throne could lead to war.  Trying to temper his nephew’s militant zeal, Hendrix sits his nephew down with their captives and explains how his wife died while they were stationed on the world of Aarde, guarding a slip to Thonios.  Aarde  Hendrix was tired of the war. It had been years of maneuvering and posturing but little fighting, but that fighting had been fierce and bloody. He and his wife, Alecia, members of the same Quarry, found themselves stationed on the island of Vrede, on the world of Aarde. The entire Quarry had celebrated being posted there.  Aarde was a water\/earth world, a splattering of islands strewn across a deep, blue ocean that spanned the planet. Some islands were huge, but none so large as to be a continent. Strategically the world wasn’t important, but Vrede held a well-known slip to Thonios that had to be guarded.  Hendrix leaned against a palm tree, his feet splayed out on the sandy beach in front of him. Alecia sat across from him. Near the shore, the ocean’s crystal clear waters merged into a deep blue, with fishing boats in the distance and not a cloud in the sky. The air smelled of saltwater.  Hendrix only had eyes for his ‘Lecia. He had taken his shoes off and intertwined his feet with hers. Their commander always gave them leave on the same day. They had been down to the village, a collection of sturdy, one-story brown wooden buildings nestled among the palm trees. Hendrix had never learned the language, but Alecia had thrown herself into it, endearing herself to the locals. She had no trouble negotiating a great price for two bottles of akaat at a large, open-air market. She had become the Quarry’s unofficial ambassador to the Aardans.  Hendrix sipped his akaat, a sweet, runny custard, with just enough alcohol to relax him.  “I don’t know, ‘Lecia. It’s beautiful here, but I’ll take a stone roof any day. Give me a hammer and chisel and a hunk of marble and I’ll carve true beauty for you.”  Alecia smiled and rubbed one of Hendrix’ feet with her own. “I admit the open sky is tough, but I’m getting used to it. I’d never seen a sunset before. And you know how I get when I see a sunset.” Her foot slid up his leg.  Hendrix giggled and a bit of akaat squirted out of his mouth before he could cover it with his hand. He set his drink down and his left hand reached out to her foot. And then his right hand grabbed her other foot and he pulled her closer while she laughed.  An arrow ripped through her throat.   Sia, Billy, and Harris sat there, staring in horror. Hendrix had a haunted look to his eyes, not meeting anyone’s gaze.  “She was in agony, dying in front of me, trying to pull the arrow out. When the second arrow came, it missed me, but I was well-trained. By the gods, I was well-trained. She couldn’t live. I didn’t have my bow. I ran back to warn our Quarry. She saw me run. I let my wife die alone because that was my duty.”  Not Middle-Grade  I didn’t mean to write the above. It just flowed from my fingers as Hendrix spoke. I flinched when I saw the words, but they felt right. That’s when the book morphed from middle-grade to young adult. I’ve had to go back and adjust ages and dialogue as a result.  Note that in the excerpt, the younger Harris wants to scrupulously follow the law and turn in Sia and Billy. It’s the older, perhaps wiser, Hendrix, who is considering breaking the law (and his oath as a soldier). If there is a central theme to the novel, it’s the constant tension between legal and illegal and right and wrong. This is not by accident.  I’m at 63,000 words, but I have little spare time. I’ll return to this precious gift to my daughter someday. Perhaps I’ll have a custom copy printed off for her. Imagine that! A sole copy of a book, written just for you, my angel.  My daughter, however, insists I try to get it published.  One day, perhaps, one day.  ","title":"The Reluctant Prince: An Unfinished Novel","url":"\/blog\/the-reluctant-prince-an-unfinished-novel.html"},{"body":"  I’m on a writing website, Critique Circle  . I want to improve my fiction writing. Periodically they used to post “writing prompts” to give you a chance to practice writing something different.  The prompt was to “create a dialogue between an apprentice trying to learn a unique magic system with a master.” The following is my entry, which took first place.   The mouse sat in the middle of the table, nibbling at the corn.  “Make the mouse disappear.”  The apprentice placed his hands over the mouse and then pulled them back quickly. The mouse was still plainly visible. The apprentice sighed.  The master asked gently, “Why did it fail?”  “Because you didn’t have any reason to believe.”  “Exactly. No one will believe an Illusion of a cliff trying to eat you . . .”  “But they’ll believe an avalanche from that cliff,” completed the apprentice.  There was a sound of glass scraping over stone, followed by a crash. The master’s head whirled around. Seeing nothing, he looked back and the mouse was gone. The apprentice was beaming.  The corn wobbled and a bite appeared. The mouse faded back into view. The apprentice let out a cry of despair and lowered his head into his hands.  “Don’t feel bad,” said his master. “If you had also hidden the corn, the Illusion would have worked. It was quite well done.”  The apprentice sat up a little straighter. “But it’s so hard to always figure out how to create a realistic chain of events on the fly.”  “Practice,” said the master, putting the mouse back in its cage. “Practice.”  He pulled out a heavy dagger and set it on the table. “Now to drive the point home, so to speak.”  The apprentice stared dubiously at the dagger when his master started speaking in low, soothing tones.  “I know this is hard. I know it’s frustrating. You can end all of this pain now if you just gently, gently push the dagger into your throat.”  The apprentice stared at the dagger and, as if it was acting of its own volition, he saw his hand pick it up and slowly start to raise it.  “No!” he shouted, slamming the dagger down on the table. “What the hell was that?”  “Language. Watch your language. And that was a Glamour. It leads you to believe things. Why do you think it didn’t work?”  “Because I didn’t want to kill myself?”  “Exactly. A Glamour, like an Illusion, can only take someone where their belief will already take them. Were you already suicidal, I’d be before the tribunal explaining why I had a dead apprentice.”  The apprentice gulped, realizing just how close to death he had come.  The master patted his hand and said, “I can’t make you do anything you wouldn’t be inclined to already do. Like Illusions, it’s a subtle art. Getting your enemies drunk can help, but I couldn’t, for example, make you want to strangle your mother.”  “You’ve never had an argument with my mother.”  The master burst out laughing. “Fair point!”  The apprentice shifted in his seat, looking at the dagger. “But if magic is so subtle, there’s not much we can really do with it, is there?”  The master reached into his robes and pulled out a deck of cards. “Watch, and learn one of the most powerful tools we have at our disposal.”  He spread the cards on the table, face up. He then scooped them up and flipped them over, face down. And then he started cutting and shuffling the cards, passing them back and forth between his hands in intricate patterns. He fanned them out.  “Pick one, and don’t tell me what it is.”  The apprentice reached for the cards, his hands moving back and forth, and pulled a card out of the deck. It was the three of clubs.  “Will you forget the card you’ve chosen?”  “No.”  The master again shuffled the cards and instructed the apprentice to shove the card back in the deck. More shuffling ensued, followed by placing the cards on the table.  “Turn over the top card.”  The apprentice turned over the three of clubs. He stared in shock. He examined the card closely before turning over the rest of the cards and seeing they were all different. There was no other three of clubs in the deck.  “But,” stammered the apprentice, “how? I know it’s not possible, but I can’t see through your Illusion. That’s not how magic works!”  The master then spent the next half hour explaining false cuts, forcing cards, misdirection, and other ways of deceiving the person watching you with a deck of cards. And then, as if plucking the card from behind the apprentice’s ear, the master held up the three of clubs again. “It takes much practice, but the charlatans in Vegas have mastered the arts of deception. Learn these, convince your enemy that they’re all tricks, and they’ll start believing in your ‘impossible’ Illusions. Even the Vegas charlatans have something to teach us.”  The apprentice sank back in his chair, deep in thought. Finally, he looked up. “The charlatans are paid well for their tricks, yes?”  “The best charlatans are quite wealthy.”  A sly smile appeared on the apprentice’s lips. “We go to Vegas. They already expect the impossible. We’ll be the most famous magicians of all time. We’ll never have to beg for our supper again.”  The master sighed. “I’ve thought of that, but sadly, magic works on minds, not cameras.”  ","title":"Flash Fiction: Charlatans","url":"\/blog\/flash-fiction-charlatans.html"},{"body":"  My eight-year old daughter surprised me again. She likes getting simple math problems to solve at bedtime and for a recent bedtime story, the equation I asked her to solve was this:  $$3\\times\\sqrt{25}\\times5$$  After a pause, she answered 75. I was happy, but then I asked her how she solved it.  For myself, after reducing $\\sqrt{25}$ to 5, I solved 3 x 5 and then 15 x 5.  I then thought 5 x 5 followed by 3 x 25 is easier because multiplying by 25 is easier for me than multiplying by 15. Hence my wanting to know which way she chose to solve it. And her answer?   Three times five is fifteen. So we have fifteen times five. That’s twelve times five, or sixty, and I added fifteen.   Well, damn. Perfectly correct. She’s a reincarnated Babylonian mathematician.  The Babylonians used base 60 math (which they inherited from Sumerians) which is why we have 60 seconds in a minute and 60 minutes in an hour. So naturally my daughter’s comment made me think of the Babylonians and I started doing some reading.  If you’ve ever worked with different mathematical bases, you might be wondering how the Babylonians memorized 60 different symbols for their numbers.  Turns out they didn’t. They only had two symbols, a unit symbol and a 10 symbol (so, some hints of base 10). We can’t easily write cuneiform, but if we pretend the units symbol is | (the pipe character) and the tens is «, they we could write 54 as «««««||||.  In Unicode’s interpretation of cuneiform (assuming you have the font installed), that’s 𒐐𒐘. But note that cuneiform is complicated and evolved over millennia, so the Unicode representations don’t appear to quite match the real ones I’ve seen online.  But where does base 60 come in? Well, recall that in base 10, the number 365 can be thought of as this:  $$3\\times10^2+6\\times10^1+5\\times10^0$$  For the Babylonians, they might have written |||||| ||||| (a six and a five with a space in between) which is:  $$6\\times60^1+5\\times60^0$$  Or in cuneiform, 𒐚 𒐙, or after 300 CE: 𒐚𒑱𒐙.  You might ask how they dealt with zero. For example, the number 1 (one) might be 𒐕, but the number 60 (1x60) would also be 𒐕 unless there was a space after it to denote the units position: 𒐕𒑱. Apparently the Babylonians didn’t think this a problem and, in fact, that middle '𒑱' character to separate digits was only introduced about 300 CE, so prior to that, you’d have to pay very careful attention to the position of the numbers in columns to hope you interpreted ambiguous numbers correctly.  That separator character solves all sorts of grief, I might add. The number 61 is 𒐕𒐕, but the number 2 is 𒐖. It’s easy to get them wrong unless you write 𒐕𒑱𒐕.  That being said, after mentioning to my daughter that Babylonian numbers were base 60, she immediately wanted to know about them. So I showed her the basics and after converting our year, 2019, to Babylonian, we got 𒌍𒐗 𒌍𒐝 (33;39). I then worked with her to show her that in different bases, the “columns” of numbers of powers of the base (from right to left, they’re 0,1,2,etc.), so after she translated the above “digits” to 33 and 39 respectively, I showed her how that works out to $33\\times60^1 + 39\\times60^0$. And that equals 2019.  Her mind was blown, but she thought it was one of the coolest things she’s ever seen.  You can read more about Babylonian numbers here.   It should also go without saying that I’m hardly an expert on cuneiform of Babylonian mathematics, so if you or or know anyone who is, I’d love corrections.  And if you’re software developer, here’s a quick Perl script I wrote to convert base 10 numbers to Babylonian:  #!\/usr\/bin\/env perl use strict; use warnings; use utf8; binmode STDOUT, ':utf8_strict'; foreach my $num (@ARGV) { my $orig = $num; if ( $num &lt; 1 || $num != int($num) ) { die &quot;'$num' is not a positive integer&quot;; } my $base = 60; my $space = '𒑱'; my %numbers = qw( 0 ␣ 1 𒐕 2 𒐖 3 𒐗 4 𒐘 5 𒐙 6 𒐚 7 𒐛 8 𒐜 9 𒐝 10 𒌋␣ 11 𒌋𒐕 12 𒌋𒐖 13 𒌋𒐗 14 𒌋𒐘 15 𒌋𒐙 16 𒌋𒐚 17 𒌋𒐛 18 𒌋𒐜 19 𒌋𒐝 20 𒎙␣ 21 𒎙𒐕 22 𒎙𒐖 23 𒎙𒐗 24 𒎙𒐘 25 𒎙𒐙 26 𒎙𒐚 27 𒎙𒐛 28 𒎙𒐜 29 𒎙𒐝 30 𒌍␣ 31 𒌍𒐕 32 𒌍𒐖 33 𒌍𒐗 34 𒌍𒐘 35 𒌍𒐙 36 𒌍𒐚 37 𒌍𒐛 38 𒌍𒐜 39 𒌍𒐝 40 𒐏␣ 41 𒐏𒐕 42 𒐏𒐖 43 𒐏𒐗 44 𒐏𒐘 45 𒐏𒐙 46 𒐏𒐚 47 𒐏𒐛 48 𒐏𒐜 49 𒐏𒐝 50 𒐐␣ 51 𒐐𒐕 52 𒐐𒐖 53 𒐐𒐗 54 𒐐𒐘 55 𒐐𒐙 56 𒐐𒐚 57 𒐐𒐛 58 𒐐𒐜 59 𒐐𒐝 ); my @result; do { unshift @result =&gt; $num % $base; $num = int( $num \/ $base ); } while ( $num &gt;= $base ); unshift @result =&gt; $num if $num; my $babylonian = join $space =&gt; map { $numbers{$_} } @result; print &quot;$orig is '$babylonian'\\n&quot;; } __END__ =encoding utf8 =head1 NAME babylonian.pl - print Babylonian numbers =head1 SYNOPSIS $ .\/babylonian.pl 123456789 𒐝𒑱𒌍𒐕𒑱𒌍𒐗𒑱𒌍𒐗𒑱𒐝 =head1 DESCRIPTION Given a positive integer, prints the Babylonian version of it. See also: https:\/\/mathandinformatic.wordpress.com\/numbers-and-number-theory\/  ","title":"Babylonian Numbers for 8-Year Olds","url":"\/blog\/babylonian-numbers-for-8-year-olds.html"},{"body":"  In the US, I have lived in Texas, Louisiana, Washington state, Oregon, Alaska, and Hawaii. I’ve also lived in Japan, the UK, the Netherlands, and now France. I grew up in serious poverty, I have been homeless, and I have seen the world. There’s nothing like experiencing other cultures first-hand and understanding what a weird, wonderful bunch humanity is.  London, United Kingdom, 2007  The following is from memory, not an old post, so some details might be off.    The River Thames with the Houses of Parliament Source    My brother Greg and his friend Tom had bought tickets to introduce me to football, the most popular sport in the world. It was for a minor team I had never heard of. Despite the minor team, there was a major police presence. Hoolaganism was the order of the day and the police were having none of it. I was warned not to stray to close to the fans on “the other side” and, indeed, the crowds were restless. As we were leaving the match, the police were making their arrests. The British take this sport a wee bit too seriously.    One of my brothers, Greg.   Greg, Tom, and I are making our way back to the Tube, I glance in a doorway and see a scene that could have been out of an art film. I stop and let Greg and Tom walk ahead as I take in the view. A lovely pub courtyard, with an old man nursing a pint lecturing a young boy. Could have been his son. Or grandson.  The man was warning the boy about the dangers of the world, the [insert list of racial slurs here] ruining England, and how everything is going to hell.  We all have hills we’re willing to die on and my hill is named “Fuck Bigots.”  Greg and Tom are far ahead of me now, but I march into the courtyard, straight to the table with the old man and the boy. Like committing a crime of passion, I have a reaction, not a plan. The courtyard is deserted save for the two at the table staring at me in surprise. I ignore the man and look at the boy. I am alone and this man looks like he has had led a rough life. This is not an intelligent thing to do.  I say, “I’ve traveled to and lived in many countries and there are wonderful, beautiful people everywhere. The world is an amazing place. Don’t let this man’s hatred poison you.”  The look of surprise on their faces turns to shock. Neither of them say a word. I wonder if my American accent, nice clothes, or apparent lack of fear caught them off guard. No one follows me as I turn and walk away, catching up with Greg and Tom.  I still think about that boy, wondering if he understood. I wonder if that boy remembers me. I hope he’s doing well and gotten beyond an old man’s hate.  Brussels, Belgium, 2017    Brussels, Belgium Source    February 3rd, 2017. I am in Brussels to speak at a conference. I hail a taxi at the Brussels airport and Leonid Brezhnev is behind the wheel. Or at least someone who looks like him.  Taxi drivers are a mixed lot. Some like to talk. Some don’t. Mine does. He was born in Belgium in a small town near the German border in the 1930s. I ask him how he came to be living in Brussels.  Back in 1958, he went to Antwerp to study at the Colonial School  . After six months of study, he was sent to Burundi to help oversee their colony. Shortly after his arrival a new doctor arrived and, the moment he saw her, he immediately decided to be sick. Two of their six children were born in Burundi. They left for Brussels in 1962 when Burundi regained their independence.  He was so inspired by his wife’s career that he returned to university and earned a Master’s Degree in business administration. He went on to be the general manager of a large company and after several years in that role, he was unceremoniously fired when a German company bought it. He had a hard time finding work at his former level and refused to “lower himself,” despite his children urging him to take anything he could find.  And then his wife died and he had no choice. He became a taxi driver, but was always filled with shame when he would see a business person he used to work with. He’d hide his face and hope they wouldn’t see him.  On that day he is 81 years old and the shame is long gone. He drives two days a week just for something to do and only takes airport passengers because they’re much more interesting to talk to than fares around town.  Grasse, France, 2022    Grasse, France (own work)   I have dropped off our car to have the brake pads changed. As I wait for an Uber on a deserted side street, a long-haired blond gentleman walks up to me, all smiles. I am wary.  (All conversation is in French)   Him: Lovely day!  Me: Yes, it is. Not a cloud in the sky.  Him: Ah, you’re English!  Me: American, actually.  Him: You must be from Los Angeles.   &lt;blink&gt;   Me: Texas, actually.  Him: You’re from Houston!  Me: No.  Him: You’re from near Houston?   Am I? Hard to say. What’s “near” in this context?   Me: Actually, no.   I’ve had weird conversations like this before. Usually they’re from someone eager to chat with a foreigner, pleased that they can communicate. However, from time to time, you experience this in tourist areas from people looking to take advantage of tourists with too little sense and too much money. They quickly shift to suggesting places to visit, offer to carry your bags, or maybe recommend a “friend” who can give you a cheap lift. Caveat emptor.  But that can’t be the case. I’m in Grasse, a small town in the French countryside. Tourists don’t go out of their way to visit Grasse (though they really should).   Him: Are you a tourist?  Me: No. I live here.  Him: OK. Bye.   And he walks away.  Leipzig, Germany, 2022    Leipzig, Germany Source    Later that week I arrive at the Nice airport in southeast France. I am very early, as I tend to do. I was heading to Leipzig, Germany, to speak at another conference. I’m grateful I arrived early because while there are no lines for other airlines, Lufthansa doesn’t have any quick check-in kiosks and it takes me over an hour to check in. While I’m in line, the lady behind me starts chatting about the line and the wait. Small talk.  Later, as I’m boarding the first leg of my flight to Frankfurt, I find I’m sitting behind her. We talk some more and she’s going to Leipzig, too.  At the Frankfurt airport, we pass the time having a quick drink and it turns out that she retired as a Russian ballerina after 22 years. She speaks seven languages and is learning to fly a private jet as part of her new job. She chats with everyone around and effortlessly switches between German, French, English, and Russian. I take her word on the Finnish, Italian, and Estonian.  During conversation I mention that my father, Jim Poe , lived in Moscow in the 1970s and I hadn’t been able to get more information about his time there. She replies that she has contacts and she’d love to help. We swap business cards.    We eventually get to Leipzig. She asks if I have a ride and I explain that my ride just sent an email apologizing for having to cancel. She insists we split a cab, but as she exits, she pays the entire fare, telling me not to worry about it.  She leaves the cab and tips the driver handsomely. As he is driving me to my hotel, already having been paid, he decides to take some side streets to show me some of the magnificent architecture in Leipzig. He is very enthusiastic about it. I have no idea what he’s saying. He points out buildings left and right. He doesn’t speak more than a couple of words of English. I suspect my accidental sightseeing tour is fantastic, but his enthusiasm makes up for the lack of understanding.  Even though I don’t have to pay, I offer a tip which he refuses. He hands me my bags with a smile and with a hearty and happy something or other , he waves and drives away.  Leipzig, Germany, the next day  I am downstairs in the hotel restaurant the next morning, searching for coffee for my room. Apparently I can’t have a pot delivered, but I can get a cup and carry it up. A young lady sitting at a table calls out to me.   Her: Excuse me! Sir!  Me: Yes?  Her: Can you help me?  Me: Uh, sure. I’d be happy to.   She stares at me. I stare back. Awkward silence. I’ve missed something.  An NPC laughs in the background. The lady looks at the NPC and turns back to me, blushing.   Her: I mean, can I help you?   I notice the pin on her lapel. She works here.   Me: I want to get some coffee for my room.  Her: You have to pay.  Me: That’s fine.   Someone leans over and whispers in her ear.   Her: Never mind. Go ahead.   So I do. The coffee machine doesn’t work.  Châteauneuf Grasse, France, Now  As I write this, my daughter is sitting on the couch, reading. My wife is working hard. I’m only 55 years old, but I’ve lost much of my hearing and my sense of taste due to surgeries. I should have them back within the year, pending more surgeries. Today is also the birthday of someone I used to know and I only learned about her passing after doing some digging when I noticed she stopped posting to Facebook.  Time is passing and I’m keenly aware of it. I’m not old, per se , but I’m getting older. My beard is grey and the hair on the top of my head is just starting to follow. Within a two or three decades, I’ll probably be gone and these words will be forgotten, along with my life. Is it narcissism that leads me to write these entries? I like to think that I’ll be leaving something for my daughter to remember me by. Immortality by proxy. It’s why I write my personal stories .    A small statuette my great grandmother bought in South America   There is, on my desk, a small figurine my great-grandmother Atwood picked up in her travels to South America. It was handed down to my grandmother. Not having a great relationship with my mother, my grandmother gave the statue to me before she died. I’ve told my daughter that it’s hers. An immortality by proxy for her great, great grandmother Atwood, though in name only. I have no stories for her.  All these stories lost forever. All of these wonderful, confusing people spending a brief moment of eternity with one another.  ","title":"Adventures in Traveling","url":"\/blog\/adventures-in-traveling.html"},{"body":"   “Hit the skeleton with the sword, take his ring, and then walk north and touch the altar.”   The year is 2022 and 35 years later, I still remember that damned sentence. The year was 1987 and I was writing a text adventure in BASIC for the COCO 2  .    The TRS-80 Color Computer 2 Source    Because it included BASIC for free, and because that was the only programming language I knew at the time, that’s what I was coding it in. If you’ve never seen BASIC, here’s a hi-lo game programmed in it  :  10 PRINT TAB(34);&quot;HI LO&quot; 20 PRINT TAB(15);&quot;CREATIVE COMPUTING MORRISTOWN, NEW JERSEY&quot; 30 PRINT:PRINT:PRINT 100 PRINT &quot;THIS IS THE GAME OF HI LO.&quot;:PRINT 110 PRINT &quot;YOU WILL HAVE 6 TRIES TO GUESS THE AMOUNT OF MONEY IN THE&quot; 120 PRINT &quot;HI LO JACKPOT, WHICH IS BETWEEN 1 AND 100 DOLLARS. IF YOU&quot; 130 PRINT &quot;GUESS THE AMOUNT, YOU WIN ALL THE MONEY IN THE JACKPOT!&quot; 140 PRINT &quot;THEN YOU GET ANOTHER CHANCE TO WIN MORE MONEY. HOWEVER,&quot; 150 PRINT &quot;IF YOU DO NOT GUESS THE AMOUNT, THE GAME ENDS.&quot;:PRINT 160 R=0 170 B=0:PRINT 180 Y=INT(100*RND(1)) 200 PRINT &quot;YOUR GUESS&quot;; 210 INPUT A 220 B=B+1 230 IF A=Y THEN 300 240 IF A&gt;Y THEN 270 250 PRINT &quot;YOUR GUESS IS TOO LOW.&quot;:GOTO 280 270 PRINT &quot;YOUR GUESS IS TOO HIGH.&quot; 280 PRINT:IF B&lt;6 THEN 200 290 PRINT &quot;YOU BLEW IT...TOO BAD...THE NUMBER WAS&quot;;Y 295 R=0:GOTO 350 300 PRINT &quot;GOT IT!!!!!!!!!! YOU WIN&quot;;Y;&quot;DOLLARS.&quot; 310 R=R+Y 320 PRINT &quot;YOUR TOTAL WINNINGS ARE NOW&quot;;R;&quot;DOLLARS.&quot; 350 PRINT:PRINT &quot;PLAY AGAIN (YES OR NO)&quot;; 360 INPUT A$:IF A$=&quot;YES&quot; THEN 170 380 PRINT:PRINT &quot;SO LONG. HOPE YOU ENJOYED YOURSELF!!!&quot; 390 END  Not too complex, but you’ll notice a couple of easy-to-miss GOTO statement which make flow control a touch odd. Still, that’s what we had to work with.  What’s worse is that BASIC was an interpreted language. Today, many languages are compiled directly to machine code, such as C. Some languages, like Perl, are compiled to byte code before they’re executed. BASIC was interpreted and executed line by line. If you had a 1,000 line BASIC program and a syntax error on line 793, you often wouldn’t find out unless that line executed. It made development hard , but it did mean that we were constantly manually doing QA on our programs, something that is often skipped in today’s TDD world.  And if that’s not bad enough, trying to optimize BASIC programs often meant shortening variable names so that they would take up less memory!  Ignoring my whining about how hard life was back then, for my text adventure, I wanted a sentence parser that was better than Zork, Adventure, and other games, but before I could parse the sentence, I had to lex it (break it into tokens). In the world of BASIC, this was no easy task.  When writing a parser, you first lex the data into discrete tokens (words, in our case) and parsers take those words, apply a grammar, and figure out what the sentence is (or if it is grammatically incorrect).  Naturally, the 1987 me had no idea what that meant. I just knew that I needed to convert those words into numbers that I could use with an ON X GOSUB command to dispatch to the correct subroutine. The subroutine would read subsequent words off of the stack to figure out what to do. I’d figure out the hard stuff later.  But the hard stuff was lexing the string into tokens.   “Hit the skeleton with the sword, take his ring, and then walk north and touch the altar.”   When I wrote my lexer, it worked flawlessly. But it took eight seconds to lex that string into the proper numbers.  Eight seconds. I remember that very distinctly because it was at that moment that I knew my dreams of writing a grand text adventure were dead if I couldn’t get that working. All of my work mapping out the hotel you woke up in and the puzzles you would have to solve were for nothing if I couldn’t fix this.  So I fixed it.  I knew that machine code would run much faster than BASIC, though I didn’t know how much faster it might be. I was just assuming it would improve the performance.  My roommate had an assembler program for the computer, along with a manual. I pored over them and because BASIC has such a paucity of features, it was not too difficult to convert the lexer to assembler, though it took me a few days.  But I had to call this routine from BASIC. That’s an odd combination of CLEAR (reserve memory), DATA (define your program), POKE (put your program in memory), DEFUSR... (define the user routine used to call your program) and USR... (call the user routine) commands. Here’s a small program to print out the ASCII value of whatever character the user typed ( from the user manual  ) (warning: pdf):  5 CLEAR 25. 12000 'RESERVE MEMORY 10 DEFUSR1=1200015 CLS 20 FOR I = 1 TO 28 &quot;STORE EACH BYTE OF OBJECT CODE 30 READ B: POKE 12000 + I. B 40 NEXT I 45 'HERE IS THE OBJECT CODE 50 DATA 173,159,160.0 60 DATA 39.250.128.10.38.12 70 DATA 173.159,160.0.39.250 75 DATA 129. 65, 45. 2 80 DATA 128. 64, 31.137,78 90 DATA 126.180. 21111 99 'TELL BASIC WHERE THE ROUTINE IS 100 POKE 275. 15: POKE 276. 211 110 A = USR1(0) 'CALL THE SUBROUTINE AND GIVE RESULT TO A 115 IF A =13 THEN END 120 PRINT &quot;CODE =&quot;; A 130 GOTO 110  Doing this for my assembly code was daunting, but I got it working. And it ran blindingly fast! I went from eight seconds to only the slightest of pauses. It was incredible! It was a miracle! It was wrong! It was giving me the wrong numbers.  I printed out the assembler on the dot matrix printer and laid out probably eight to ten meters of paper (my memory is probably faulty), got on my hands and knees with a pen and started laboriously tracing through that code, line by line. This was called a desk check  and was one of our primary debugging techniques in the 80s.  Hours later, with sore knees (and with my roommate claiming I was showing off), I found my bug. I fixed the program, reran it, and it was a success. I had a lexer for simple English sentences and could move on to my parser.   I never finished that game. It was my roommate’s computer and he moved away. By this time I was learning C at Lower Columbia College  , Except that there was a big problem. This was the first C class offered at LCC and we didn’t have a C compiler. We had a Prime Minicomputer  we could use, but Garth Conboy  , who was writing the compiler, hadn’t finished it yet. So our first few C programs for class were written out in long-hand and submitted to our professor who would manually check them for correctness (desk checks again!).  I still remember the day that Professor Little brought in his computer setup and demonstrated one of the programs his company produced. It seemed awesome. Someone asked him about the sales of the program and Professor Little started to explain that it was very popular. It was so popular, in fact, that pirates were copying his program and redistributing it for free.  And then our professor broke down crying in front of all of us. He was teaching at LCC because so many people were using free copies of his software instead of paying for it that his company was going broke. It was a humbling experience for all of us to watch our professor cry.  Later, when we got the C compiler, the first version was buggy. Some expected commands didn’t work and, one day, while debugging a problem, I added a printf statement and the program worked! I eventually just printed an empty string and that was enough to get my program to run. I had to show it to my professor and explain the problem to ensure he wouldn’t mark me off on it.   Fast forward to 2022, 35 years later, and I still think about that BASIC program. Eight seconds to lex that string. I had gotten to the point where I could walk about a map, pick up and drop items, but that was all. But damn, eight seconds! And between me being inexperienced (I had only started programming in ‘82), and having to use an assembler, it took me two weeks (as I recall) to get the lexer working at an acceptable speed.  What would it take today? The following is a rough hack, but it shows (sort of) what I was doing back in 1987.  #!\/usr\/bin\/env perl # Less::Boilerplate is a personal module I use for quick # hacks, but it makes my Perl more &quot;modern&quot;. You can read that as # &quot;use less boilerplate&quot; or &quot;useless boilerplate&quot;. Both are correct. use Less::Boilerplate; use Benchmark 'countit'; my $words = get_words('\/usr\/share\/dict\/words'); my $sentence = 'Hit the skeleton with the sword, take his ring and then walk north and touch the altar.'; say countit( 8, sub { parse_sentence( $sentence, $words ) } )-&gt;iters; sub parse_sentence ( $sentence, $lookup ) { # this is not perfect, but it replicates when I did back in the 80s. my @words = split \/\\s+\/, lc $sentence =~ s\/[[:punct:]]\/\/gr; my %numbers; foreach my $word (@words) { if ( my $number = $lookup-&gt;{$word} ) { $numbers{$word} = $number; } else { warn &quot;$word not found&quot;; } } return \\%numbers; } sub get_words ($file) { open my $fh, '&lt;', $file; my %words = map { state $i = 1; lc($_) =&gt; $i++ } split \/\\n\/, do { local $\/; &lt;$fh&gt; }; return \\%words; }  That consistently prints out a number around 1.9 million. In other words, this quick ‘n dirty hack that I wrote in a couple of minutes replaced two weeks worth of work in 1987 and ran almost two million times faster.  That’s right. This trivial example was not only easier to write, but it was two million times faster than the BASIC code I wrote 35 years ago.  You kids have no idea how easy you have it.  ","title":"Programming in 1987 Versus Today","url":"\/blog\/programming-in-1987-versus-today.html"},{"body":"  If you have an interest in math, by now you may have hear that the mathematician Po-Shen Loh has found an easy way to solve quadratic equations. Those equations looked like this:  $$ax^2 + bx + c = 0$$  And they were every student’s nightmare in algebra class, because when factoring the equation didn’t work, you learned this solution:  $$x={-b\\pm\\sqrt{b^2-4ac} \\over 2a}$$  Finding the solution for many quadratic equations was a difficult chore, especially if the answers involved imaginary numbers. Thanks to Loh, it’s now easy. Unfortunately, many of the sites I’ve visited seem to have rather cumbersome descriptions of this relatively easy solution. So to help myself remember the solution, and thus help you learn the solution, let me skip the proof of this novel solution  and just show you how to solve it. If you wish to know how it works, there are many sites on the web which will get into the details. Just hit your favorite search engine for something like “Po-Shen Loh quadratic equation”.  You’ll only need basic algebra to understand this.  Step 1: Getting x 2  Loh’s solution requires that the first term, $x^2$, is always just $x^2$, and never multiplied or divided by anything. So we don’t want the first term to be $3x^2$ or $x^2 \\over 4$.  That’s fine, because we can just multiply or divide, as needed, so get to $x^2$. For example, if we start with this equation:  $${x^2 \\over 2} - 3.5x + 6 = 0$$  We can just multiply by $2$ to get:  $$x^2 - 7x + 12 = 0$$  Step 2: Getting $-b$ and $c$  Next, we need to isolate the opposite of $b$. Since quadratic equations are in the form $ax^2 + bx + c = 0$, and we have $x^2 - 7x + 12 = 0$, we know that the opposite of $b$ is $7$ (since it’s $-7$ in the equation).  Next, we need $c$. In the equation $x^2 - 7x + 12 = 0$, we can see that $c = 12$.  The key insight in Loh’s work is that we’re searching for numbers which add to $-b$ and multiply to $c$. Thus, for our example:   Sum ($-b$) is $7$  Product ($c$) is $12$   If you remember the above, you’ll remember this in no time because that’s the key to making everything work.  Step 3: Finding the roots  So we need two numbers which add to $7$ and multiply to $12$. Rather than guessing, we start with the addition.  First, we know that two numbers, added together, will equal any number if their average is half of that number. So let’s take half of that $7$, or $3.5$ (or $3{1 \\over 2}$ if you prefer fractions).  So we know that the two numbers are (using $u$ for a placeholder) are $3.5 - u$ and $3.5 + u$.  Since multiplying them gives us $c$, or $12$, we have:  $$(3.5 - u)(3.5 + u) = 12$$  Using the difference of squares, left hand side becomes $3.5^2 - u^2$, or $12.25 - u^2$. So we have:  $$12.25 - u^2 = 12$$  And that gives us:  $$.25 = u^2$$.  Taking the square root of each side gives us $.5 = u$. Since we that the numbers are $3.5 - u$ and $3.5 + u$ we know that the solutions are 3 and 4.  You can substitute those numbers into this equation to check the answers:  $${x^2 \\over 2} - 3.5x + 6 = 0$$  Simplifying  Skipping over many mathematical steps, let’s take this:  $$4x^2 - 8x - 16 = 0$$  We want $x^2$ as the first term, so divide each side by four: $x^2 - 2x - 4$.  We know that $-b$ is $2$ an $c$ is $-4$, so we have $(1 - u)$ multiplied by $(1 + u)$ = $-4$.  $$1 - u^2 = -4$$ $$5 = u^2$$ $$\\sqrt{5} = u$$  So the roots are $1 \\pm \\sqrt{5}$.  Let’s do it again  Let’s use this equation:  $$x^2 - 2x - 24 = 0$$  We know that $-b$ is $2$ an $c$ is $-24$.  So half of $-b$ is $1$, we have:  $$(1 - u)(1 + u) = 24$$  That reduces to:  $$1 - u^2 = 24$$  Which becomes:  $$25 = u^2$$  So we can see that $u$ is $5$ and $(1 - u)$ is $-4$ and $1 + u$ is $6$, giving us the roots of the equation. See how simple that is? It takes practice, though.  A harder example  Let’s take this equation:  $${x^2 \\over 2} - x + 12 = 0$$  We need $x^2$, so let’s multiply both sides by $2$, giving us:  $$x^2 - 2x + 24 = 0$$  We know that $-b$ is $2$ an $c$ is $24$.  So half of $-b$ is $1$, we have:  $$(1 - u)(1 + u) = 24$$  That reduces to:  $$1 - u^2 = 24$$  Which becomes:  $$-23 = u^2$$  So $u$ is $\\sqrt{-23}$, or $i\\sqrt{23}$.  Thus, the two possible solutions are $1 \\pm i\\sqrt{23}$.  That’s not a fun problem to solve with the traditional quadratic equation solution.  If you want more practice watch Loh’s video on the topic.     ","title":"The Easy Solution to Quadratic Equations","url":"\/blog\/the-easy-solution-to-quadratic-equations.html"},{"body":"  My Muse is back. Unfortunately, she’s not a very good one. I imagine her sitting on a barstool with too much makeup, chipped nail polish and an unlit cigarette dangling from her lips. She bats asterisks of mascara at me and if I light her cigarette she’ll push a pen in my hand. It seems she’s only good for one-night stands or at best a weekend fling. Period.  And what would I write? Maybe I’d pen “Emma”, but I’d make her a prostitute and ruin the story. Or “Romeo and Julian”, the Capulets and Montagues sitting around, fast friends, with wife-beaters crawling up their beer-engorged guts and muttering that them faggots ain’t natural.  My Muse would be aghast, her chins quivering in disgust.  And the fruit of my labor? Two published books, both on computer programming. My first article in print? “Logic Programming in Perl,” published in the now defunct The Perl Review. Numerous technical articles on the web. I love what I do for a living, but my Muse keeps sneaking back and pinching my bottom for another flirtation with writing fiction.  Don’t get my wrong. I do write fiction. I wrote a screenplay, have an unfinished screenplay, an unfinished novel. A few short stories. All of it unpublished.  I fear my long-standing fling with my Muse has conditioned me to accept what I do have rather than what I could have. I mock her, but she knows how to show me a good time if I only let my guard down. She’s an escritorial fuck buddy I’m unwilling to introduce to my friends.  I think of writing as a dream. I also think of it as a supreme act of narcissism. Of vanity. Who am I to think that others will want to read what I write? And yet here I am, perversely, writing. I’ve stolen a quickie from my Muse. No wonder she doesn’t stay long. Why should she satisfy me if I don’t satisfy her?  Sometimes we go for long walks together and we plot our novel. How is this different from daydreaming? Writing is almost respectable, but until I set pen to paper or fingers to keys, even toying with respectability eludes me.  I’m a daydreamer who has succeeded by finding himself competent in a field he enjoys. And it’s not even that I’m the best in my field. I’ve worked hard to learn more, but harder to market myself. I get hired not because I am an astonishingly good programmer. I get hired because I am Ovid. Because my ability to write, though not noteworthy, is nonetheless more interesting than a few others in my field and people remember me.  I am known for my writing, not my technical ability. Despite this I rely on the latter and not the former. I know if I pay attention to my Muse, perhaps even trouble to learn her name, she’ll reward me. She’ll strip off her makeup, her fickleness, and open herself to me. Perhaps I’d be poorer and happier and writing about missing the technical challenge of programming, but at least I would know.  Until then, my writing will reflect my Muse. Sickly sweet perfume and trying too hard to please. My writing, though cloying, reflects how I think when the Muse is nearby. I’m a saccharine Faulkner. I’m artificial. My Muse and I grunt and groan and sweat and thrust and play and writhe and finally lie back in exhaustion, each secretly fearing that we’ve let the other down. And I still don’t know her name.  ","title":"Why I Write","url":"\/blog\/why-i-write.html"},{"body":"    Early Life  The Soviet Years  Valerie  Brenda  The Children  Credits  Update 1  Update 2     This is a eulogy for my father because his story should be told. Much of this story is rambling and hard to follow. This is because I’m grieving for the father I never had and because no one in our family knows what happened in this man’s strange life. If you’re confused, so are we.  And if you have any stories to share of Jim’s life, please share them in the comments. Corrections are also welcome because I’m quite certain that some of this is dead wrong.    James Lewis Poe, April 22nd, 1941 -June 26th, 2019. RIP.   My father, Jim Poe, has passed away. He died on the morning of June 26th, 2019 and this is the story of his life as I know it. I’ll try to be fair, but there are several unreliable narrators in this story, not the least of whom is me. Further, most dates in this story should be taken with a grain of salt because it’s very hard to piece them together. All of Jim’s known surviving children, and his fourth wife, Valerie, have contributed to this recollection.  Early Life  This writing is an attempt to process my grief for a man I did not know, but who I always wanted to have as part of my life. And I have, to my astonishment, a plethora of heretofore unknown brothers and sisters who are similarly grieving. To all of his children who’ve survived Jim (Gayle, Greg, Lewis, Lynne and possibly others we don’t know about), I love you dearly and forgive me for airing dirty laundry. And for all our father’s lies, I do believe him when he denies being a Soviet spy. More on that later.    Jim and my wife, Leïla.   Jim was born in Texas in 1941.   He has a surviving brother who lives in Colorado.  When Jim was 16 years old (though given his numerous “birth dates”, this is questionable), he married for the first time, to a 13-year old girl. This was legal at the time and given that she was with child, understandable. Their child, Randy, was my half-brother. I first learned about his existence when I was about 12 or 13, when my mother, Jim’s second wife, came outside to tell me that Randy was dead. Apparently he went into the woods with a gun and never came out. Hunting accident? Suicide? Murder? I don’t know. I just know that, at that age, I was confused I had never been told I had another brother.    Jim Poe, December, 1965   After Jim had left his first wife, he talked wistfully about a woman named Janis who he briefly dated in Texas. Jim being Jim, they drifted apart and over the years, he said he wondered what happened to her. In 1967, the year I was born, he found out. Janis Joplin  became a superstar. That being said, Jim is one of the many unreliable narrators of this story.  Jim met my mother, Caroline “Karee” Tom around 1961, in Portland, Oregon. They fell in love and had a quick, private wedding, much to the consternation of my maternal grandparents. In 1962, my sister Gayle was born. A couple of years later, our brother, James Lewis Poe Jr., was born, but he passed away at six months, due to a heart problem from birth. I was born in June of 1967, in Fort Worth, Texas, the town our father had fled from.  I use the word “fled” with care because I don’t know exactly what happened. Jim told me what happened, but he was very drunk at the time and later denied all of it. Other family members have shared variations of the story, but with it all being hearsay and drunken recollections, I don’t want to tar his name with a story I don’t know to really be true. I do remember, however, Jim being sober and telling me he joined the navy “one step ahead of the boys in blue” (the police). That would fit with every variation I’ve heard.  In the early 60s, Jim was a sailor and was briefly in Portland, Oregon, where, as mentioned, he met my mother. He had already left his first wife and my mother dropped out of college, at least in part to be with him. They traveled quite a bit and in 1966, while he was in the Navy stationed in Gourock, Scotland, I was conceived. My sister Lynne, née Charlotte, was also conceived in Scotland to a different mother, Eliza.  Lynne was conceived two weeks later than I. I like to describe it as an embarrassing exercise in familial mathematics. Lynne just says we’re twins. I think Lynne’s description is nicer.  Our brother Greg, same mother as Lynne, was born February, 1969. And later our brother Lewis was born April, 1978, to Valerie Poe, Jim’s fourth wife (between Eliza and Valerie was Janath, but I don’t know much about her).   If you’ve been keeping track that makes seven children by four mothers. That we know of. Jim denied there were any more, but given that he’s often been less than honest with us, we’re skeptical.    Our sister, Gayle, was the only sibling I knew growing up. She remembers Jim as a wonderful father who would sometimes have strange people visiting our home in Scotland, including a man with one hand. My paternal grandmother, Birdie (who I didn’t know, but Gayle did), recollects that Jim had a solid career in the Navy and had received a commendation for helping to capture a Soviet spy onboard ship. This was the first of many strange “Soviet” stories surrounding our father. However, I can’t verify this because even though I filed a Freedom of Information request for Jim’s naval records, I was informed the US Navy had no records for him.  That being said, Gayle shared this tidbit:  I remember him picking me up like a sack of potatoes and putting me over his shoulder when he would come home. I don’t like peppermint lifesavers, but I loved them because he gave them to me.  She also said:  I remember dad coming home one night and a man was with him. He had a bandage on the end of one arm. I, being 4 or 5, thought he had lost a hand. I also remember mom throwing a metal file box at him and screaming at him. They left. I questioned him about that when I was staying with him in Idstein (Germany). He remembered that but said I was mistaken about the missing hand.    My father’s ship, USS Simon Lake, Holy Loch, Scotland, 1969   It was during this time in Scotland that the relationship between Jim and my mother deteriorated, as evidenced by my sister Lynne’s birth two weeks after my own. There’s much more I can say, but I don’t know what is true and what is not.  My mother returned to the US (possibly involuntarily) and I was born in Fort Worth. Jim left the Navy at this time and applied for many jobs in the UK, hoping to stay. But he had to return to the US and when he first saw me in early 1968, he picked me up and I vomited all over him. I was a precocious child.  As Jim tells the story, he was in the US, but quickly received a job offer in the UK and he was on the next plane back, never to live in the US again. I was told at one point that mom had been contacted by the FBI regarding Jim’s whereabouts, with instructions to contact them if he ever returned to the US, but I’ve no idea if this is true. Many years later Jim would return to the US to do some work, but he never lived there again and had no desire to do so.  And that’s when the story gets even more muddled. After he returned to the UK, he fathered Greg and, later, visited London for another job interview. When he returned home, he found his wife, Eliza, in a compromising situation and ordered her to sort herself out (I’m omitting some details). She did so by leaving Lynne and Greg with a neighbor and moving away. The children were then put up for adoption.    By one version of the story, he tried to claim his children but couldn’t since he was an American and not married to Eliza. But when I asked him about he said he gave them up because he didn’t have a lifestyle conducive to having children and he had no idea how to be a father. However, he was married again someone named Janath and, according to Lynne ...  [Jim probably gave Lynne and Greg up for adoption because] Janath instigated it, he paid maintenence to the local authority for our upkeep until we were adopted and sent presents at Christmas time.  Now, if you think the story was muddled before, it gets even weirder from here on out.  The Soviet Years  So Jim was with Eliza and our brother Greg was born early in 1969, but from 1970 to 1974, Jim was living in Moscow and met and married someone named Janath. Jim worked for a UK company, ICL (International Computers Limited)  , as a computer engineer, but behind the Iron Curtain  . This led to quite a bit of familial speculation about him. He was reluctant to discuss this and when he once sent me his “resumé for his life” (which I’ve sadly lost), he was fairly detailed about many of his jobs outside the Iron Curtain, but was quite fuzzy on what he did while he worked in the Soviet Union and Czechoslovakia. I do know that in the US Navy, he held a top-secret clearance as a Polaris Missile technician. Given that he was working for a British company behind the Iron Curtain, that would certainly upset the US government.  By Jim’s account, he was persona non grata at the US embassy while he lived in Moscow. Despite being fluent in Russian, he couldn’t exactly hang out with Russians because the Soviet government frowned on fraternizing with Americans. Thus, Jim’s “watering hole” was a bar at the British embassy because there was no one else with whom he could socialize. Here are a few tidbits that I remember from my conversations with Jim (paraphrased from memory, of course):  I was in the British embassy, having a drink, when I asked the waitress her name. She said it was “Janeth”. I was surprised because that’s such an unusual name and I told her that I knew a woman named “Janath”. I thought she’d find that interesting but instead, she got very upset. Her name was “Janice” and she though I was making fun of her lisp.  Jim would also tell a story of being at the British Embassy in Moscow and George Best  , was there. Best is considered to be one of the finest football (soccer) players in history and was also a raging alcoholic with a huge fan following in Moscow. He and Jim got drunk together, with Jim allegedly being the last man standing.   Jim had many talents and drinking was one of them.    And then there was the late night “drunken” email, that I asked him about later. He sent it to Lewis, Greg, and myself—but not to Gayle or Lynne—in 2010. I assume he was drinking at the time (he usually was) because he rarely shared personal anecdotes like this:  This movie - “Quiet days in Clichy” was cult when I was in Moscow in 1970 to 1974. All of the embassies had a copy and played it regularly. I got the tape from my friends Tony Conyers  of the Daily Telegraph and Dennis Blewitt   from the Daily Mail. It really brought back some fond memories when I found this on YouTube.  It is from a book by Henry Miller. “Clichy” is a suburb of Paris.  The link was to a preview of “Quiet days in Clichy” and, from what I understand, even though they played it at the British embassy, it was banned in the UK. I was very surprised by the email because our father rarely shared anything personal. He would sometimes share a joke, but this was part of his life. So, since I work extensively with the Web, I tracked down the film and create a private link he could download it from. And I watched it.  It was porn. I was happy that my father was reaching out, unprompted, to share something about himself. And it was porn.    A safe for work trailer for the movie “Quiet Days in Clichy.“.   After I sent him an email with the download link he replied “Thanks Curtis, You can take it down.”  And that was it. He sent the email at 11PM his time, apparently drunk, trying to reach out.  I’ve no idea what to make of this, other than knowing that he had feelings, but didn’t know how to express them. I asked him later and his “fond memories” were watching this banned film, with friends, in the British embassy in Moscow. From what we can gather, Moscow was a fantastic time in Jim’s life, full of screwing around, partying, and generally being Jim. It was a time of his life he seemed wistful for, though he did seem to suggest that he only had the British embassy bar as an outlet.  You have to keep in mind that this was during the Cold War between the USSR and the USA. Given that Jim worked on Polaris missiles and was now living in Moscow, speaking fluent Russian, the US government was not happy. Jim said it was laughable the few times he was approached by someone for casual chitchat and it would be obvious that US intelligence was again trying to find out if Jim was a traitor. Jim swears he wasn’t a spy and his being there wasn’t political. It was just a job.  Valerie  Lewis’ mom, Valerie, is an amazing woman and I’m happy to have met her. She was kind enough to share more details of Jim’s life.  A few years later, in 1975, Jim, still working for ICL, was living in Prague, Czechoslovakia, when he met Valerie, the mother of Lewis Poe. As Valerie tells it (and shared with permission):  I first met Jim in the British Embassy club bar in Prague and thereafter a couple of times in the Prague office of ICL for which UK company we both worked. We then ‘got it together’ and he arranged to work there on a more permanent basis so we could be together. After a few months we transferred back to the UK where he was sent on a training course and I worked in the centre of London. We lived together in north London and bought a flat in Harpenden before we moved to Germany. Lewis was born about 3 years later.  I was followed and stopped in Germany when I was with a friend as at that point my car still had Czech number plates. The Bader Meinhof gang who were active then were trained in Czechoslovakia, so I guess they were suspicious. A week or so later Jim and I were followed in Rudesheim by someone who made it obvious enough it was a warning of some kind. But I can assure you we were not involved in anything secret, underhand or exciting! We just fell into the category of suspicion as we had both worked in Eastern Europe and my politics were quite left of centre at that time. Cloak and dagger might have been fun!  She also writes:  One story which is possibly funny. Lewis was about 18 months old and we had been to see close friends in Holland who supplied Jim with a small amount of hash. Just before the German border (which we normally drove through without any check) we decided as an extra precaution to put the hash between 2 diapers which Lewis then wore. We got stopped and searched very thoroughly and just as I thought we were done for Lewis woke up and started to scream. Good old Lewis did the trick. They soon told us to go! I laughed all the way home.  So at this point, it’s 1979 and Jim is living with Valerie in West Germany. Jim sometimes had to work in East Berlin and he described passing through the infamous Checkpoint Charlie  , the only gate between West and East Berlin that foreigners were allowed to use.    Checkpoint Charlie, 1963. Source    As an American traveling to East Berlin on business, it was natural that he was going to be questioned. He told of handing his passport over and explaining Ich spreche kein deutsch (“I don’t speak German”), but apparently he said it so well that the East German officer didn’t believe him. The officer started getting upset and raising his voice with Jim. So Jim turned to the Soviet officer behind the East German officer and said in fluent Russian, “Would you kindly explain to this man that I am not kidding and I don’t speak German?”  The Soviet officer said nothing, but he smiled. The East German, however, upon hearing Jim speak Russian, immediately stood back. Jim said his passport came flying back to him like a frisbee and he was waived through, no questions asked. After that time, whenever it was that guard, Jim would smile and wave and the guard let him pass, never stopping him again.  And then the story takes a darker turn, as it often seemed to do with Jim.  When Lewis was around 3, Brenda became Jim’s secretary and he started his affair with her. He made life untenable for us telling me one day on the phone to get out as he would not be responsible for his actions if he found Lewis and I there when he got home. I have to say that I did love him very much when we were together. We had so many good times and he was truly happy about Lewis and used to tell me how blessed he was to have us both. When he met Brenda he turned from a loving husband and father to a bit of a monster. Sorry this probably isn’t what you need or want to know but unfortunately it was very much part of him.  Brenda was much younger than Jim and, I think, a little over ten years older than myself. Remarkably, Jim stayed with Brenda from 1981 until she passed away in 2006 from cancer.  Brenda  This is to honor Jim’s life, but doing so without saying a bit about Brenda would be to dishonor Jim’s memory, as she was the final love of his life. They planned to retire together in Greece, but she tragically passed away from brain cancer in 2006.  When I was living in Amsterdam in 2001, I took a train to Germany to meet Jim for the first time. We talked on the phone first, and he explained “Brenda isn’t much older than you and she’s very beautiful. When you’re here, I want you to think of her as your mother .”  This was Jim, being Jim. He was apparently so used to men chasing after any woman they wanted that he was concerned I’d make a move on his partner! I wasn’t happy he said this, but I wanted to meet the man.  When I arrived at their home in Idstein, it was a lovely, four-story home. And on one wall was a beautiful Keith Haring  artwork. I walked up to examine it and realized, with shock, that it was a painting, not a print.  Keith Haring paintings can sell for millions of dollars. Here was my father and Brenda, living in a lovely four-story home, with million-dollar paintings on their walls.  Except it wasn’t. Brenda was a forger, but a legal one. When she saw art she liked, she would simply paint a copy. It turns out that she previously worked for the Bank of England and her team would investigate forged banknotes. If the notes were good and the forger was caught, Brenda’s team would interview the person to find out where they obtained their ink, their paper, how they made the notes, and so on. Then they would replicate the notes and try to pass them to banks—always with a “get out of jail free” card in their pocket if they were caught.  For ten years after she left, whenever there was a major forgery in the UK, she’d get a visit from British police, verifying that she was not involved.    Jim also wrote the following about her work on Lotus Notes:  Brenda is doing fine. She’s as bogged down with work as I am. Her Lotus Notes Domino system is taking Europe by storm. She’s sold it to a sister division here - Abbott International “AI” (they opened a tender and then blind testing on 3 selected vendors. Brenda’s system won on both performance and price). They are the pharmaceutical side of Abbott. It’s revolutionised the German Affiliate where she works. IBM, who own Lotus, are taking a big interest and have committed to support if anything should happen to Brenda’s developer and his company. They are really doing some pioneering work with Domino and have a direct line to IBM support for any issues which come up. The developer has been made an IBM approved 3 party provider with guarantees backed by IBM. He’s laughing all the way to the bank.  She was an amazing woman. Sadly, when she passed, she wasn’t married to Jim,   so under German law, Jim had no rights to their shared assets unless he could prove what he paid for. A family member of Brenda’s claimed everything and Jim was left almost penniless. It was so bad at one point that Lewis was being sued for thirty thousand euro in “overpaid” child support because many payments were made from Brenda’s account. Fortunately, that was apparently dropped, but not before the lawyer suing Lewis tried to get Lewis to pay the legal fees.  Jim later was hired as a well-paid contractor for Abbott Pharmaceutical, alleviating his money woes for a while, but he never recovered financially, or emotionally, from Brenda’s death.  Brenda was indeed a lovely woman. And for all of Jim’s brusqueness, it was offset by her grace.  The Children    Your author and his sister, Gayle   Given all of that and Jim’s penchant for walking away from relationships or contacting his children, you might wonder how we all came to know one another. As with all things “Jim”, it’s complicated.  In 1987, I was living in Washington state, in the US, and had an odd hobby, skip tracing  . I did it for fun, helping friends find lost buddies and the like, but I realized one day that I might be able to find my father. It turned out to be remarkably easy and a few hours later, I was on the phone with a heretofore unknown cousin. Her father was Ron Poe, Jim’s brother. My cousin was telling me that Jim would send Christmas cards every year and she was kind enough to read off Jim’s address in Idstein, West Germany. Then Ron entered the room, asking his daughter who she was speaking to. He then took the phone and we chatted briefly. Ron wasn’t aware that I now had Jim’s address and told me that he hadn’t heard from Jim in years.  Though annoyed at the time, I later realized that Ron had no idea I was who I claimed to be on the phone. It’s only natural that he should protect his older brother. I’ve always wondered what he thought he might be protecting his brother from, though I have my suspicions.  Armed with an address, I was delighted to discover that West Germany also had an “information” telephone number to call to get the phone number for a given person and address. It was just after 5 PM local time, so when I called Jim, I managed to reach him just after 2 AM. Brenda answered the phone and after I explained, she handed the phone to Jim. I had finally found my father. His very first words are forever etched in my mind.  “How did you find me?”  He didn’t seem happy to hear from me, but I wrote it off to the shock of a long-lost child waking him up at 2AM.  After I got off the phone, I called my sister, Gayle, to let her know that I found Jim. I gave her his address and phone number and made her promise to wait until morning in Germany before calling him. Gayle promised. After we hung up, she promptly broke that promise, calling Jim immediately. It was not a good night for him.  Over the next few years I would call a few times, but Jim never seemed anxious to talk. It wasn’t until 1996 that I thought to ask an obvious question: do Gayle and I have any other brothers and sisters? Jim seemed very uncomfortable with this and finally said “David.”  At this point you might be wondering if you missed something because I’ve not mentioned David once, and for good reason. Lewis’ first name is David, but no one calls him that. Jim knew it because Lewis and Valerie lived with Jim and Lewis had gone to Germany and stayed with him on a holiday more than once. So me searching for “David Poe” wasn’t likely to turn anything up.  When I asked for more information, Jim told me about Valerie and where she might live in the UK: Southampton or Poole, on the southern coast. He claimed he didn’t have an address or phone number. This, I later learned, was a lie. Further, Jim did not mention Lynne or Greg.  I spent the next three years trying to track down Valerie and “David” and eventually found an address for Valerie. Unfortunately, her phone number was ex-directory (unlisted). Since I knew nothing about her, I was afraid that she might throw away any letter I sent, so I kept trying to find her phone number. And one day, in 1999, I was experimenting with software named Alphaworld  (one of the first online 3D worlds) and mentioned my tale of woe to another Alphaworld citizen. He was from the UK and told me to stay online. Half an hour later, he returned with the phone number of Valerie’s neighbor.      Lewis Poe   Very excited, I called the number and immediately ran into a new stumbling block. The woman I spoke with knew Valerie, but her son was Lewis, not David. Further, the woman insisted that no, Lewis didn’t have any brothers or sisters. I had to explain a couple of times that we had the same father, but different mothers and Lewis probably didn’t even know about me (it turns out he did but couldn’t find me). Eventually she said she’d go ask and a short while later, I was speaking to Lewis on the phone.  I later obtained a passport and flew to the UK, meeting him for the first time. We got along fabulously. And it turns out that Valerie’s awesome and there wouldn’t have been a problem with me sending that letter.    Greg Baddely   While visiting them, they informed me that there were two more siblings, but they’d been given up for adoption. They didn’t even know the genders. So I spent a few years trying to track them down. British adoption laws are strict, so there was no way I could get the information that way. The aforementioned Freedom of Information Act request for Jim’s records was, in part, to find any clue of who else he had known. But I was at a dead end. I finally gave up.  And then in January 28, 2005, Jim forwarded an email to Gayle and myself (which I forwarded to Lewis):  Jim - Hello from the past. I came across this person looking for you while surfing the net(at least I think it’s you she’s looking for). I keep in touch with [REDACTED] so had your e-mail address. Thought I would forward this to you rather than give her your address. Hope I didn’t F.U. How have you been? Would like to hear from you. Don’t know if I ever thanked you for pulling me out of the water at Lake Marion as I was going down for the third time. - Wes    Lynne Moore   Included in that email, was Lynne’s name and email address. When she got home from work that night, she had email from Gayle, Lewis and myself, brothers and a sister she never knew she had. She said she broke down and started crying.  Though it was kind of Jim to share that with us, Jim was still Jim. In his first email to Lynne he asked “How do you feel your life will be enhanced by knowing me?”  By a strange coincidence, I was being recruited by several British companies to help fix their IT systems. About a year and a half later, I had moved to Nottingham, UK, about an hour’s drive from Stoke-on-Trent where Lynne and Greg lived. Later, I moved to London and Greg and I shared a house, and later a flat together. I live in France now, but all the surviving brothers and sisters get on quite well. It’s the best part of Jim’s legacy.  Jim wasn’t all bad. When Lynne needed her car repaired, Jim bought her a new one. When Leïla and I got married, he contributed nicely to the wedding. He never understood what it took to be a father, so when he had money, he spent it on us from time to time, perhaps to make up for his past.  I also know he left behind many friends in Idstein, Germany. They’re great people and many of them are heartbroken he’s passed. Jim was never a good father, and clearly he was often not a nice man. But he’s all I had for a father, even if I didn’t know him well. Jim, wherever you are, I hope you’ve found peace.    Jim Poe, Curtis Poe, and Jim’s grand-daughter, Lilly-Rose.   Credits  I would like to thank my sisters, Gayle Poe and Lynne Moore, my brothers Greg Baddely and Lewis Poe, Lewis’s mom Valerie Poe, and my cousin Naomi Clements for helping to fill in some details. Thanks to my wife, Leïla, for proofreading and suggestions. There are many stories left out, some by request and others because they would cast living people in a bad light. Any errors and omissions, of course, are mine.  Update 1  Amongst other things, it appears that Jim lied about his birthdate. We the date as April 22nd, 1939. He was actually born April 22nd, 1941. There had been some earlier rumors that Jim might have lied about his age in order to join the Navy. If he was “one step ahead of the boys in blue”, as he put it, he might have had a compelling reason to lie, a lie he maintained for the rest of his life. Now that I’ve seen a birth certificate, albeit one issued in 1957, it confirms that he lied. It also raises some questions about his age when he was first married.  We’ve also found ourselves in the most delightful Catch-22. Germany won’t release a death certificate without a certified (apostille) birth certificate. Texas won’t issue that birth certificate because, now that they have been told he’s dead, the certificate can no longer be released unless it’s stamped “deceased,” for which we apparently need a death certificate.   Update 2  We’ve resolved the sitation with Jim’s death certificate (the US Consulate in Germany was very efficient in helping us), and Jim’s ashes are now buried and a tree should grow from it in the years to come. And in a parting gift, it turns out we may have one or two more siblings in Russia. . Not, I should say, a surprise.  ","title":"A Eulogy for My Father","url":"\/blog\/my-father.html"},{"body":"    Introduction  Astrobiology  Phosphine on Venus  Implications     Introduction  By now, if you’re a science geek like me, you’ve probably heard the news that Venus shows possible signs of life  . Yes, hellish Venus, of all places, might harbor life. This is due to the detection of significant amounts of PH 3 (phosphine) in the atmosphere. While the original paper downplayed the possibility that possibility that this phosphine was generated by life, nonetheless, it’s worth understanding the excitement and what it could mean.    The planet Venus.   Astrobiology  For many, Venus doesn’t seem like a likely candidate for life to exist. The surface temperature is 450&deg; Celsius (842&deg;Fahrenheit—hotter than Mercury!). The atmospheric pressure at the surface is 90 to 95 times greater than on Earth. What’s more, on the surface the atmosphere is overwhelmingly carbon dioxide and nitrogen. But you wouldn’t suffocate. You’d burn and be crushed to death first.  Despite that, the upper atmosphere of Venus is the most Earthlike in the solar system, with even NASA discussing ideas for exploration or colonization  .     So what was so interesting about the recent research paper?  In biology, there’s a subfield called astrobiology. I like to think of astrobiologists as science fiction writers with advanced degrees in biology. They consider the possibility of life elsewhere in the universe, what it might be like, and how we can detect it. One of the things they frequently discuss is biosignatures  . These are chemical signatures which, if they exist, suggest some form of unusual chemical process that might indicate life. For example, large amounts of free O 2 (oxygen) in the atmosphere would be a huge biosignature, as oxygen reacts with just about everything and is quickly removed from the atmosphere. While there are physical processes which can cause oxygen to remain free in the atmosphere, we have so much free oxygen in our atmosphere because life produces it. The detection of large amounts of it in an alient atmosphere would certainly be a great indicator that further investigation is warranted.  Methane is another biosignature gas because its existence is usually associated with life. We’ve been discovering lots of methane on Mars  and we don’t know why. Given that the 1976 Viking landers possibly detected metabolism on Mars  , the strange discovery of methane on Mars merits closer investigation. We don’t know what’s going on with Mars and life, despite it being very unlikely, could easily explain what we’re observing.    The planet Mars.   However, it cannot be stressed enough:  If the question is “aliens?”, the answer is almost certainly “no.”  Seriously, stop thinking it’s extraterrestrial life. Every time we’ve gotten our hopes up, they’ve been dashed. And that’s a good thing: it means science is working; we’re not searching for fun, we’re searching for truth.  Phosphine on Venus  Now, about that paper  .  The paper is fairly easy to follow, even if it’s aimed at a technical audience.  The authors of the paper were aware that phosphine is known as a powerful biosignature  . While phosphine can be produced in energetic reducing atmospheres such as Jupiter and Saturn (and has been detected there), rocky planets such as Earth and Venus shouldn’t have significant phosphine in their atmosphere, though Earth does due to phosphine production by extremophiles. However, we can potentially detect phosphine on exoplanets, so the researchers decided to use Venus to create a benchmark for further studies, but they found ”unexpectedly, our initial observations suggested a detectable amount of Venusian PH3 was present.”  This was a surprise. So the paper focuses on two areas. First, they considered all the possible ways that whatever they detected was not phosphine. After eliminating all sources of confusion they could think of, they concluded that, yup, it was probably phosphine.  The next part of their paper asks the big question: what could cause this?  They considered many, many different ways in which PH 3 could be created, but there simply isn’t enough free energy on Venus for that to happen in any significant quantities. For what little PH 3 could exist, they considered the lifetime of the molecule and outgassing, again finding an unsustainable amount of PH 3 . So where does it come from?  Here’s an important section of their research.   Even if confirmed, we emphasize that the detection of PH 3 is not robust evidence for life, only for anomalous and unexplained chemistry. There are substantial conceptual problems for the idea of life in Venus’s clouds—the environment is extremely dehydrating as well as hyperacidic. However, we have ruled out many chemical routes to PH 3 , with the most likely ones falling short by four to eight orders of magnitude To further discriminate between unknown photochemical and\/or geological processes as the source of Venusian PH 3 , or to determine whether there is life in the clouds of Venus, substantial modelling and experimentation will be important. Ultimately, a solution could come from revisiting Venus for in situ measurements or aerosol return.   In other words:   The PH 3 (phosphine) is not strong evidence of life  There could be previously unknown chemical processes producing it  It would be difficult for life as we know it to survive in Venus’ upper atmosphere  We know of no natural (non-living) processes which can produce this phosphine on Venus   There’s actually been plenty of prior discussion about the possibility of life in Venus’ clouds, with Carl Sagan writing about this in the 1960s. Serious research articles on the topic appear regularly  and pop-sci articles love to describe scientists pointing out that dark patches in Venusian clouds could be micro-organisms  .  Implications  While it’s very unlikely, what does it mean if life is found on Venus? That would depend on its origin. If we found that Venusian life had the same DNA or RNA as Earth life, that would strongly suggest a common origin. The panspermia  hypothesis, long laughed at in science, is looking more and more credible all the time, particularly since we find Martian meteorites on Earth. Wouldn’t it be fascinating if life originated on Mars and made its way to both Earth and Venus?  While I love that thought, there’s the more exciting possibility that potential Venusian life shares little, if any, biological similarity to Earth life. If so, that would suggest a separate origin. The implications of that would be staggering.  Currently, we only know of one place in the entire universe where life was created: Earth. If we had two independent creation events in the same solar sytem, this would imply that it’s common enough that the universe should be teeming with life. Absolutely teeming. As our technology improves, it would be easier to detect those biosignatures on the thousands of alien planets we’ve discovered. It would be amazing.    The Pioneer Plaque Source    As for intelligent life elsewhere, that’s a different story, but if the universe is teeming with life, that tremendously raises the chance for intelligent life elsewhere.  I can imagine few scientifics endeavors more important than sending new missions to Venus and Mars to investigate one of the most fundamental questions humanity faces: are we alone?  Update : new research indicates that the alleged phosphine detected in Venus’s atmospher is probably sulfur dioxide. Here’s the paper  and here’s a news article about it  .  ","title":"Life on Venus?","url":"\/blog\/life-on-venus.html"},{"body":"    Summary  History  Marx  Words Mean Things  The Nordic Model  Types of Socialism  Utopian Socliasts  Ricardian Socialism  Marxist Socialism  Democratic Socialism  Social Democracy  Social Market Economy  Possibilities     Disclaimer : I do not believe in Marxist socialism; I don’t have a problem with a social safety net, however.  Recently on Facebook—words which often do not bode well—a friend of a friend was not happy that his personal definition of socialism wasn’t accepted as a universal definition:   The government owns and controls the means of production; personal property is sometimes allowed, but only in the form of consumer goods.   Because his personal definition wasn’t accepted, he couldn’t create strawmen arguments to knock down, nor could he meaningfully engage in debate because, gosh , it was simply unacceptable that the definition he just Googled wasn’t good enough for the other debate participants.    The Old Port of La Rochelle Source    Except that this definition, for what it’s worth, is the one that is largely accepted today in the US. Meanwhile, in La Rochelle, a lovely medieval town on the West coast of France (one in which I used to live), they’re expanding the harbor to provide better services to billionaire’s super-yachts. It’s socialists who voted to do so because La Rochelle is undeniably a socialist town. Turns out the socialists here in France often have no issue with private property or money, but they strongly object to the abuse of either.  For Americans, it gets even more confusing when self-described “democratic socialist” Bernie Sanders says “thank God” for capitalism.     So what gives? Do socialists agree that private property is OK or that it should be abolished? Can’t have it both ways, can you?  Summary  Because, sadly, modern people’s attention spans for reading are short, I’ll share the definition of socialism now and you can decide if you’ll continue reading or not.   Socialism: a political or economic system that promotes social welfare.   That’s it. And it’s not exclusionary. If we look at the social democracies known as the Nordic Model, they are very pro-capitalist and promote business welfare, too.  So how do we get from government controlling the means of production to this definition? That takes a bit of explaining.  History  You can’t have a meaningful discussion of the meaning of socialism unless you understand the origin of the idea. And no, Karl Marx didn’t come up with it. His most famous work, Das Kapital  , was published in 1867, but it was fifty years earlier, possibly with the publications of work by Robert Owen  , that socialism was gaining traction. However, you can’t even appreciate this without context.    Adam Smith Source    In 1776, Adam Smith published The Wealth of Nations  , a book today regarded as one of the first books describing modern economics. And honestly, it had to be. The foundations of modern economics are land, labor, and capital. Historically, land wasn’t freely bought and sold. It was largely held by nobility who would no more think of selling their ancestral properties than you would think of selling your parents. The concept of laborers freely seeking the best possible wage also didn’t exist. In the middle ages, most people were serfs, tradesmen, clergy, or nobility. Fiefs, lands to which serfs were tied, still existed in the 17th century  . And utilizing capital to leverage your wealth to build even more wealth wasn’t really a thing because modern economic thought was, well, not thought of. In fact, the innovative possibilities of capital combined with profit-seeking labor is very much a modern invention.  Adam Smith came along when economies were transitioning from zero-sum Mercantalism to industrialized capitalism. With the birth of the industrial revolution in the mid-1700s, we started to see a magnificent growth in profits. We also saw a magnificent growth in new forms of human misery. In the paper entitled Time and Work in 18th-Century London  , Hans-Joachim Voth cites Herman Freudenberger and Gaylord Cummins as arguing that   annual labor input possibly increased from less than 3,000 to more than 4,000 hours per adult male between 1750 and 1800.   That’s up to twice the hours most Americans work today (and Americans tend to work far more hours than most other developed nations).    Child Labor Source    And frequently those laborers were children. It appears that many children were effectively slaves during the industrial revolution  . And by “slaves”, I mean that with all of the horrifying implications. Robert Heilbroner, in his famous economic history The Worldly Philosophers  wrote:   Indeed, a recital of the conditions that prevailed in those early days of factory labor is so horrendous that it makes a modern reader’s hair stand on end. In 1828, The Lion, a radical magazine of the times, published the incredible history of Robert Blincoe, one of eighty pauper-children sent off to a factory at Lowdham. The boys and girls—they were all about ten years old—were whipped day and night, not only for the slightest fault, but to stimulate their flagging industry. And compared with a factory at Litton where Blincoe was subsequently transferred, conditions at Lowdham were rather humane. At Litton the children scrambled with the pigs for the slops in a trough; they were kicked and punched and sexually abused.   Sadly, these accounts of the horrors of child labor during the industrial revolution are not scarce. Is it any wonder that many people objected to this new idea of capitalism?  One of the most prominent objectors was the aforementioned Robert Owen. A highly successful businessman, Owen was aghast at how workers were treated and decided to buck collective wisdom and create a more humane business. In 1799, he purchased New Lanark Mills  and   Under Robert Owen’s management from 1800 to 1825, the cotton mills and village of New Lanark became a model community, in which the drive towards progress and prosperity through new technology of the Industrial Revolution was tempered by a caring and humane regime. This gained New Lanark an international reputation for the social and educational reforms Owen implemented. New Lanark had the first Infant School in the world, a creche for working mothers, free medical care, and a comprehensive education system for children, including evening classes for adults. Children under 10 were not allowed to work in the Mill.   And Heilbroner reports that from 1815 to 1825, there were over 20,000 visitors, including nobility and business owners, all trying to figure out how this mill had become hugely profitable.  Owen, along with Henri de Saint-Simon  , Charles Fourier  , and others, formed what we today call the “Utopian Socialists.” Generally speaking, they were a reaction to the brutality of the industrial revolution and these thinkers envisioned better worlds, or utopias, based on decency, kindness, and social needs. However, with the possible exception of Owen’s direct action in New Lanark, these were imagined worlds, built on dreams, not evidence.  They were, however, the first socialists, they were all over the map, and they had little to do with governments controlling the means of production.  In contrast to them were the Ricardian Socialists  . They focused quite heavily on the labor theory of value  which argues that the value of a good or service is the total amount of labor required to produce it. Yes, many people think that’s Marxist because Marx was quite fond of it, but their beloved Adam Smith championed it! In Chapter V of Wealth of Nations  , he wrote:   The value of any commodity, therefore, to the person who possesses it, and who means not to use or consume it himself, but to exchange it for other commodities, is equal to the quantity of labour which it enables him to purchase or command. Labour therefore, is the real measure of the exchangeable value of all commodities.   The Ricardian’s (so-called because David Ricardo  also accepted the labor theory of value and his arguments heavily influenced them) argued that because value is derived from labor, those who labor deserve the rewards. This is curious to me because it often implies that managers and owners who supervise and direct physical labor—often without doing it themselves—are somehow less valuable, but I digress.  To the Ricardians, labor should reap the rewards of commerce, but is instead exploited. One of these was Thomas Hodgskin  and the opening paragraph from Wikipedia sums up many issues quite nicely (emphasis mine):   Thomas Hodgskin (12 December 1787 – 21 August 1869) was an English socialist writer on political economy, critic of capitalism and defender of free trade and early trade unions. In the late 19th and early 20th centuries, the term socialist included any opponent of capitalism , at the time defined as a construed political system built on privileges for the owners of capital.   For Hodgskin, capitalism was built and maintained to serve the privileged class. However, he didn’t object to capitalism per se . He objected to what we might call crony capitalism  . He didn’t want a world where one prevailed due to government meddling and backdoor deals; he was, in fact, a Libertarian who loved the market system, but championed labor. He insisted on the rights of unions to exist and advocate for workers (because which Libertarian would object to free association?) and he strongly objected to rent-seeking behavior  . But he was definitely a capitalist, through and through. And a socialist in that he advocated for the needs of society, not just the monied class.  Marx    Karl Marx Source    It wasn’t until Karl Marx came along and skewered the Utopian and Ricardian socialists, that people started thinking of socialism in terms of Marx’s more specific view. This was, in part, because in The Communist Manifesto and other writings, Marx and Engels created a more detailed description of, and argument for, socialism. They both acknowledged the moral considerations of the Utopian and Ricardian socialists but destroyed them for being fantasies.   And given that the Manifesto called for revolution and swayed many people, Marx’s views on socialism were indelibly etched in people’s minds.  In particular, in The Communist Manifesto  Marx wrote of the Utopian socialists:   By degrees, they sink into the category of the reactionary [or] conservative Socialists depicted above, differing from these only by more systematic pedantry, and by their fanatical and superstitious belief in the miraculous effects of their social science.   But what of his own views? In the Manifesto, Communism is the result, but socialism is a necessary step:   We have seen above, that the first step in the revolution by the working class is to raise the proletariat to the position of ruling class to win the battle of democracy. The proletariat will use its political supremacy to wrest, by degree, all capital from the bourgeoisie, to centralise all instruments of production in the hands of the State, i.e., of the proletariat organised as the ruling class; and to increase the total productive forces as rapidly as possible.   And that is effectively the definition that many hurl against socialism today: naïve workers gutting honest, hard-working, private enterprise in favor of an inefficient nanny state. Or to phrase it in the other extreme: the proletariat rising up against the bourgeoisie to put power in the hands of the people.  Personally, I don’t buy either definition because a) I don’t see it happening and b) I can’t help but roll my eyes at those those think the definition of an ever-evolving word must be set in stone by an essay written 172 years ago, ignoring all modern thought on the subject.  Words Mean Things  Yes, words mean things and then they mean other things. For example  :   Girl meant any young person – “gay girl” meant girl, not lesbian, and “knave girl” meant boy – while “man” meant any person, regardless of gender.   The word “girl” was originally gender neutral and arrived in the English language around 1300 CE  , but we’re not certain of its origin.  Or consider that the word “meat”, coming from the Old English maet , simply meant “food”  and its current definition of “animal flesh” didn’t arrive until the 1300s.  Words, of course, have no intrinsic meaning. When I hear the word “peer”, I think something like “equal in quality”, but if I’m speaking French, the word « pire » has the same sound as “peer” but means “worst,” a radically different meaning.    Words have the meaning we ascribe to them and nothing more. But this causes many problems if we don’t understand that another’s definition is different from our own. For example, some people think “meat” only means pork or beef, leading to ridiculous incidents of vegetarians being served chicken because they “don’t eat meat.”  So if you have a definition of socialism that doesn’t match my own and we don’t realize this, it’s very easy to have a confused argument. So what do people think socialism means today?  According to a 2018 Gallup poll of Americans  :   &nbsp; 2018 1949  &nbsp; Sep 4-12 Sep 3-8  Equality - equal standing for everybody, all equal in rights, equal in distribution 23% 12%  Government ownership or control, government ownership of utilities, everything controlled by the government, state control of business 17% 34%  Benefits and services - social services free, medicine for all 10% 2%  Modified communism, communism 6% 6%  Talking to people, being social, social media, getting along with people 6% --  Restriction of freedom - people told what to do 3% 1%  Liberal government - reform government, liberalism in politics 2% 1%  Cooperative plan 1% 1%  Derogatory opinions (non-specific) 6% 2%  Other 8% 7%  No opinion 23% 36%   It’s interesting to note that “government control” has now slipped to second place between “equality” and “benefits and services.”  And then we have this lovely New Yorker article discussing how tangled the definition of socialism is in America today  .  The Nordic Model  Or consider the Nordic Model  . The social democracies which have adopted the Nordic model have strongly embraced globalization, capitalism, and social welfare. While they differ in varying degrees, key similarities in all of these countries include:   a comprehensive welfare state  large-scale investment in human capital, including child care and education  labor market institutions that include strong labor unions and employer associations   Depending on where you want to draw the line, the countries which follow the Nordic model are Finland, Denmark, and Sweden, but many include Norway and Iceland in this group. These countries tend to have strong economies and, most importantly, five out of the seven happiest countries in the world  follow the Nordic model. I don’t believe that is a coincidence.  Given the historical definitions of socialism, I think the social democracy of the Nordic Model fits quite well.  Types of Socialism  So now I think we can break socialism down a bit into a few different types. The following, of course, is by no means exhaustive and in some cases, is a touch at odds with definitions provided by economists.  Utopian Socliasts  Largely a reaction to the brutal industrial revolution capitalism, the utopians envisioned perfect societies, all radically different from one another. They’re generally considered to be naïve.  Ricardian Socialism  Also reacting to industrial revolution capitalism, the Ricardians accepted David Ricardo’s description of the labor theory of value and argued that all wealth belongs to the laborers. They tended to reject ideas such as rent or profit, but some, such as Hodgskin, were also capitalists.  Marxist Socialism  This is what most Americans mean when they say “socialist”, which is curious because it’s a dying breed in the West (though still somewhat popular here in France). In short, Marxist socialism is when the government owns the means of production of goods and services. Marx also called for the abolition of private property, but not personal property (consumer goods).  Democratic Socialism  In this model, democracy is paramount, but it’s combined with a drive towards a socialism closer to the Marxist ideal. There are no countries which currently follow this model.  This is what Bernie Sanders claims to be, but in reality he fits the social democrat model better.  Social Democracy  I like to think of this as “modern socialism”. There are several strands, but a key feature of Social Democracy is embracing capitalism while trying to contain its worst excesses, possibly via nationalization of problematic industries. Strong social safety nets are a key component of this. However, this leads us to ...  Social Market Economy  Though not a form of socialism in the classic sense, I feel that it fits the zeitgeist of feelings on socialism today.  This is what much of Europe has adopted today, including the economic powerhouse, Germany. It’s similar to Social Democracy, but tends to shy away from nationalizing industries. Social Market Economies tend to support the market, but not guide it, and have very strong social safety nets. They also strive to constantly understand the economy and adjust and adapt, rather than try to find a one-size-fits-all economic “Theory of Everything.”  Possibilities  Imagine a world where you don’t go bankrupt because you tripped and broke your leg. Imagine a world where you don’t have to choose between forgoing college or decades of crippling student debt. Imagine a world where you’re paid a living wage as a waiter—or you can get filthy rich building your own business empire. That’s socialism today.  And an overly simplistic review of the meaning’s evolution over time:   Pro-social and sometimes pro-capitalist (Utopian and Ricardian socialists)  Government control of production (Marx)  Pro-social (today)   Yes, there are still people who will pedantically insist that Marx’s definition is the “one true socialism”, either because they agree with it or they wish to attack it. But his definition wasn’t the first, nor will it be the last. And while economists (and pedants) still cling to Marx, many of us are looking to the future and see the potential of caring about people as much as business.  ","title":"What Is Socialism?","url":"\/blog\/what-is-socialism.html"},{"body":"   This is more of me sharing tidbits of my life so that my family can know more about my past. I don’t want to be the enigma that is my father .   The year was 1978. I was eleven years old. My family was living in a run-down complex of duplexes in Schertz, Texas, and my only friends in the complex were the McClouds. They were a couple of boys, one my age and one slightly older. They were the Platonic ideal of “juvenile delinquent” and my mother didn’t like me playing with them, but there weren’t many children in the complex and she wasn’t home at the moment.  We were playing tag and I was running from the younger McCloud, desperate to not be tagged “it.” He was chasing after me because, as the least athletic, I was the natural target. I was skinny, but I was clever. I was running towards the ivy-covered chain-link fence in our back yard. I was betting that if I could jump that fence, he would chase an easier target. As I grabbed the top rail, I vaulted and soared over the fence perfectly. I also felt a wasp nest beneath the leaves crush under my hands.    These things scare the hell out of me.   Despite not seeing it, I knew instantly what it was. I can’t convey the visceral terror that I felt at that moment. I went from being the least athletic to the most as I ran for my life. The wasps, meanwhile, hastily convened a meeting to discuss how to redress such a gross violation of their sovereignty and decided to do what wasps do. I was right to run.  As one of them reached me and stung my arm, I fell to the ground, screaming as much from terror as from pain. And I knew with a dead certainty that the wasps would be swarming over me, stinging me again and again. What happened was far worse.  The younger McCloud had also vaulted the fence. He ran up and kicked me, screaming “you’re it” as he ran off. It was not a good day.  If memory serves (and being almost five decades ago, it most certainly does not), I was only stung once, but that, and feeling the wasp nest crush under my hand, was enough to instill a deep, lifelong terror of them.  Fast forward a few years and I was spending summers on Alvin Wittig’s farm. He was a close friend of my mother and Alvin and his father were happy to have me spend a few weeks there. I’d read decades-old copies of Reader’s Digest that his father kept on a bookshelf, sometimes marveling that an article written in the 1950s would be reprinted decades later, but with a new author’s name and the dates and times updated. I’d help Alvin herd cattle. I’d mess around in his tool shop. I’d ride a demon creature from Hell, a pony named Queeny, he had rescued from a family that treated it poorly. To this day, I’m sure science has equines wrong. They’re actually hivemind creatures, all of them networked together to communicate a shared desire to kill me. Queeny used to cheerfully do a “stop-drop-and-roll” in rocky fields if I tried to ride her. She’d buck and jump and rub against trees to get me off her. Alvin laughed.  Other equines had similar attitudes towards me. The gentlest nags in the world would take off at a gallop if I mounted them, but we finally reached a compromise: I agreed not to ride them and they agreed to stop trying to kill me. My daughter loves these demon creatures from hell. She takes horse-riding lessons while I watch, with a fixed smile on my face, awaiting the moment the horses realize she’s my daughter. Fortunately, the horses are too stupid to make the connection.  Where were we? Oh yes, wasps!  No, not yet.  Alvin was also a beekeeper. To me, bees were little short of thalidomide-baby wasps, runty bullies whose only saving grace is that they would die if they stung me. There were always plenty of bees flying around Alvin’s farm and I’d swat at them and Alvin would patiently explain to me that if I left them alone, they’d leave me alone. He’d be explaining this to me, with tenderness and affection, as he’d remove yet another bee stinger from my skin. Alvin, requiescet in pace , was a good man.  Alvin would show me how to smoke the hive, remove the honeycomb, cut off the top layer and spin it in a centrifuge to remove the honey. Then he’d cut off a piece of honeycomb for me to enjoy like bubblegum. I never liked honey, but I never told Alvin that. Eventually, I got the point where I’d rest my hand at the entrance to the hives, the bees crawling all over me while they made their way in and out. They never stung me again and I found a strange sort of peace, watching this erstwhile terror go about the business of living.  Fast forward 20 years later and I found myself a barista, working a beach event where I was managing other baristas serving overpriced coffee to tourists on an Oregon beach and when the bees started swarming, our customers ran away screaming, frantically batting at the bees. I instructed my staff to start pouring circles of vanilla syrup onto paper plates and setting them off to one side. The bees happily swarmed the plates, a bobbing collar of lace absorbing the sugar. They were beautiful. When the bees would swarm again, I’d call out, “time to feed the bees,” and they’d pour more syrup for them, keeping the tourists safe from nature.  But wasps. Fuck them.  Today, I live with my wife and daughter in a house in the southeast corner of France, about 45 minutes from the Italian border. I’m blessed with a wonderful family, a paradise to live in, and wasps. Did I say, “fuck wasps”? It bears repeating.  We live in the countryside. You might even say in a forest. We like to eat outside, but this is a siren call to the wasps. Last year, we’d frequently find ourselves running inside when the wasps muscled in, delighting in the smell of our barbecue and the meat that they crave. The bees would show up, too, but they didn’t bother me as much any more. But the wasps? Fuck wasps.  I hunted for their hives to no avail. We set up a string of wasp traps around our home, killing many of these foul, evil beasts, but not stopping the invasion. There was a war on and the dominant species on the planet was losing it.  The winter bought us some respite, but thanks to the never-ending wonder of climate change, France experienced an unusually light winter. When the leaves fell last winter, we found two huge wasp nests the size of basketballs, several stories above the ground in tall trees. I wasn’t worried because those nests were dead. I was wrong. The queens that should have died in the frost did not.  This year, 2022, is being called The Year of the Wasp  .  Due to the heatwave, we’ve been leaving the windows of our home open. The wasps are now coming in for cat food. And heaven help us if we try to cook anything because the battle ensues. On my best day, I killed nine wasps. Then I went out and refilled six wasp traps with the sweet fluid that leads them to their well-deserved deaths. The wasps traps are filled with dead wasps, but refilling them is a challenge because they’re often filled with live wasps. And more keep coming.  We’re losing this war. We need to a different approach.  We could just refuse to eat outside, but we’re not giving up our quality of life for damned wasps. Even if we did eat inside, we’d have to close all of the windows. Not an option in this heat. Call exterminators and ask them to scour the forest for wasp nests? I don’t think so.  My wife came up with a different approach. Why kill the wasps? They have a right to live, too. She decided to try a peace offering. We offer a bit of food for the wasps, away from where we’re eating. After they swarm the food, we sit down to eat in relative peace.  So far, it mostly works. Our hot war became a cold war and is becoming a détente. We still have wasps dive-bombing our plates from time to time, but we’re learning to be at peace with them. It’s not easy as they still scare the hell out of the eleven-year-old boy in me, but all of us are getting more used to them. We’re becoming a wasp sanctuary.  My instincts tell me to kill them or run. My conscience tells me we’re doing the right thing.    Wasps eating our peace offering.   Last night, we ate sushi. We put aside a little platter of raw fish for the wasps and left them alone. When they settled down to eat, we ate our meal in peace.  Be safe, little wasps.  ","title":"The Wasp Sanctuary","url":"\/blog\/the-wasp-sanctuary.html"},{"body":"    The Early Days  Barista  My First Test  How I Became a Bill Collector  Sexual Harassment  Leaving     The Early Days  Note: As usual, much of the below is from decades ago and my memory isn’t perfect. In the unlikely event you are mentioned here, it’s possible your memory will be different from mine.   I am often asked how I got into writing software for a living. This is the story of my first real programming job and the strange path that took me there.  In 1982 I was attending Keystone School  in San Antonio, Texas. It was a very dark time in my life and my geometry teacher recognized that I was in trouble, so she allowed me to play in the computer room that was reserved for older students. I learned BASIC programming and discovered something in school that I loved. After I left that school to go back to a public school, I started writing (bad) games in BASIC and laboriously typing in programs I found in magazines.  I had begun with Commodore PET computers and moved on to the technological marvel that was the TRS-80 (called the “trash 80”). By 1985, I already written a fully-featured lo-res graphics program for the TRS-80 that allowed users to create images, saved across multiple files (I didn’t understand data compression back then) and stitch those files together to print the images on a dot matrix printer.  Those were the days. I had demo images with that, including a nice, but pixelated minotaur’s head, dragon’s head, and other AD&amp;D images.  One would think that after I graduated high school in ‘85 that I would go on to college and do something with my life, but one would be wrong. We were dirt poor and at the time, to qualify for financial aid for college, I had to first apply for a Pell Grant.    That required that my parents provide several years of past tax returns to show what level of financial support they could provide. My mother explained to me that my step father did the taxes. He had failed to file them for years, so no Pell Grant for me. No financial aid for me. No college for me.  The next several years were me bouncing around from odd job to odd job, mostly fast food. At one point I had worked at Wendy’s  restaurant in Anchorage, Alaska, Kingsville, Texas, Universal City, Texas, Longview, Washington, and Honolulu, Hawaii. I was a pathetic peripatetic, wandering from state to state, deeply embarrassed that the “Most Intelligent of the Class of ‘85” of Marion High School was an itinerant burger flipper.  Years later, I eventually went to a community college in Portland, Oregon and managed to get an Associate of Science degree with a 4.0 GPA   , having taken every economics class they offered with a specialization in international relations.  An Associate’s Degree is a two-year degree and unless you use it to go on to a four-year college, it’s worthless for many jobs.  Barista  Which is how I found myself working as a barista (espresso bartender) when a young, pretty lady walked up and placed her order, leaving a large tip in my tip jar. I was proud of the fact that I was the highest tipped male barista in our chain of stores and I accomplished this by flirting. So I flirted with her, in my naïveté thinking it was innocent fun. I had known her and her boyfriend as regular customers, but that’s all. And that’s when she made me a rather curious offer.  I declined, pointing out I had a girlfriend, to which she replied that my girlfriend could join us.  Er, no. So when her boyfriend arrived later to place an order, I apologized, explaining that through I didn’t know him well, he should know what happened.  He swore, thanked me, and explained that their lifestyle was a bit different. However, they agreed not to proposition people without the other’s knowledge. I still wasn’t going there, but he and I started talking more and eventually we became friends. Turns out he was a software developer and when he found out I used to write software, we started talking about this frequently.  One day he came in to place an order and said he was quitting his job and he’d be happy to recommend me. The software was written in something called UniBasic  and he was sure I could handle it. He would stay on for three months to train me.  I got the job and it got my foot in the door, but to my surprise, it was often less professional than the fast food restaurants I had worked at.  My First Test  My friend was leaving his job on the last day of December. I had been on the job for just three months, learning their software, but only being allowed to work on it part-time. Otherwise, I made sure all of the employees were turning in their paperwork correctly. On that last day, the controller called me into his office. He explained that they needed to increase the price of everything by X%, varying per category, and have it ready for tomorrow.  This wasn’t just running a bunch of SQL. We were using a UniData multivalue database  , prices were scattered all over the place, and I wasn’t very good with their query language. Worse, our license allowed us to write code to read the data, but not change it. Updating this data was done through software running on a terminal and it created audit trails for everything. I needed audit trails, too.  This was a bad idea on many fronts. First, my friend made it clear how much he despised the place and he was leaving at 5PM because he had a New Year’s Eve party to attend. He explained to me how to do the job and I knew enough about the design of our database to figure it out, but it should not have been a rush job.  Next, since software I wrote could only read data, not change it, we had a huge risk. If we changed the data and were caught, we’d have to pay $25,000 (almost $50,000 in today’s money) to buy a copy of the software outright. We were a small company and that would have hurt us. We’d also lose all technical support from the company that “designed” this software.   and we’d no longer get free upgrades. Having a few hours heads-up on New Year’s Eve that I needed to risk destroying the company’s business wasn’t fun, but this was how the company worked. So I worked.  After writing the software to update the prices, I ran it. If it worked, it would look like people had spent days manually updating all of the prices, complete with audit logs. I had done it in hours. But if I—a junior developer—corrupted the data, I might be fired. Happy F’ing New Year.  It was getting late in the evening and I was tired and stressed. I ran the software. I corrupted the data.  The only reason I didn’t get fired is that I wrote the software to terminate after the first change and I had printed out all of the data that would be altered by the first run (this seeming much easier than the alternatives in those early days of software development). The controller and I manually fixed the data, I figured out what I did wrong, and ran my software again. This time it worked.  I went over all of the data carefully. I double-checked everything I could. The database was complex and prices for different kinds of goods were stored in different ways, but I couldn’t see anything wrong with what I did. I explained everything to the controller (who, to be fair, had become a friend of mine. He was also surprised by this last minute request). I told him I could remove the code that caused the software to halt after the first change. The software would run overnight and we wouldn’t know until the morning if it worked. The controller told me to do it and get to my party.  January 1st, I get a phone call early in the morning. I’m hung over and not looking forward to this call. It seems they had been checking all of the data they could.  The software worked.  How I Became a Bill Collector  This was a small, family-owned business, with several stores offering a variety of products, many of which had to be ordered months in advance to be handmade in Europe and shipped to the US. The business wasn’t struggling financially, but as far as I could tell, they weren’t getting rich. They needed more money. In fact, I needed more money. I asked for a raise.  At the time, I was paid $x dollars an hour for my administrative role and a couple of dollars an hour more for when I wrote software, capped at 15 hours a week. This wasn’t legal, but I wasn’t slinging espresso or burgers. I never complained, but I still asked for that raise.  We had a new office manager who was not well-liked, was a rubbish manager, but knew our products like the back of his hand, having recently been promoted from warehouse manager. He listened to me, nodded, and explained that they were having trouble with customers ordering goods and not paying, leaving the company with a huge accounts receivable (AR) balance. The company simply didn’t have the money to pay me.  Oh. That’s interesting. In my prior life, I had also worked in car sales. I knew exactly what this was: a negotiation. The manager probably thought he had told me “no,” but I heard his objection and I isolated it  .  Me: “You don’t think I deserve a raise?”  Him: “Yes, you do, but we can’t afford it.”  Me: “And this is all because of the outstanding accounts receivable?”  Him: “Yup, sorry about that.”  Me: “So if I can get those people to pay, you can afford to pay me.”  He was trapped by his own words, but to get out of that, he made me an offer I couldn’t refuse, but couldn’t win. If I could get AR to zero by a certain date, I’d get a raise. He wouldn’t budge from zero. We had a lot of outstanding accounts, but not an insurmountable number. However, the goods we sold weren’t cheap and getting people to pay wasn’t going to be easy. I was allowed to take any reasonable steps to enforce this.  That’s how I became a bill collector.  The first thing I did was, of course, write software to produce daily reports for me on the next accounts I would go after, and the steps I had taken to collect on those accounts.  I sent email to everyone I could, and called the others (all customers gave us phone numbers). An astonishingly large number of those people had no idea they owed money and paid up. No one had bothered to contact them! I was surprised at how easy it was. But a large number of them simply didn’t answer.  My next step: I received permission to contract with an actual bill collector. They would buy the debt from us, paying a fraction, and then it was their problem. That’s when I discovered the world of potential clients wining and dining you to win your business. I only accepted lunch from them, but now I realize that even that is dubious.  I chose a bill collector, the company signed the contract, and another large subset of accounts were cleared. But there will still more.  One, in particular, was someone who told me on the phone that she owed nothing. But she did, and I had the paper to prove it. In fact, I often went back to the paper because if this went further, I wanted more than just ones and zeros on a hard disk. She clearly owed the money and it was a lot, so I tried to collect directly rather than hand her off to the bill collector.  Turns out she was a lawyer and immediately replied that we were going to be sued for harassment. The company agreed to write her debt off and the grounds that a trial would cause more pain than her debt. Another account cleared.  On the day of my deadline, the manager greeted me with a smile. He was very happy with the work that I had done but, unfortunately, I hadn’t reduced AR to zero. No raise.  I handed him the AR report. If the owners of the company paid their bills for the products they bought, our AR would actually be positive. Should I send them to collections?  The trap was sprung, the manager was caught. I got my raise.  Sexual Harassment  The worst nightmare at this job, though, was the sexual harassment.  Three people, Alice, Bob, and Charlie. All of them were nice people and friends of mine. I was still very young and not understanding how business was supposed to work, so I flirted with Alice. She flirted back. We liked each other, but she was also an extremely devout Christian, so my atheism was an issue.  Nonetheless, we continued to flirt. A little harmless fun, right? Bob didn’t flirt with her, but he cracked lots of jokes, was well-liked, and he and I started hanging out after work. He was a great guy; we’ll get back to him. I liked Charlie, too, but Alice didn’t. So when Charlie would make inappropriate comments to her, just like I did , she ignored them.  One day she came to me with a problem. The company was driven by testosterone-fueled cronyism and she didn’t trust upper management. She said she was being sexually harassed by Charlie and didn’t know what to do.  Oh, shit. I wasn’t entirely sure of the law, but a few thoughts crossed my mind.   I stopped flirting instantly  If I didn’t act, I could be culpable  I had to talk to upper management  She was about to be fired for being our worst salesperson   She reluctantly agreed to let me explain the issue to management, but wouldn’t tell me details. After speaking with management, they were incredibly pissed that they couldn’t fire her. However, they would take no action against Charlie because he was our number one salesperson. They eventually agreed that Alice and I would have meetings to work the situation out, with me being the intermediary between her and upper management. I was not happy about this, but that was that. So some evenings, Alice would leave the sales floor and have private meetings with me in a locked back office, trying to find a way forward. This was not professional, but I didn’t know that. And remember, I left the sales floor and locked myself in an office with a pretty woman who wasn’t selling anything but still kept her job.  Bad optics. Really bad. Bob, my new friend who I was hanging out with a lot, joked in front of Alice and myself that he hoped we were having a good time in the back office. He knew nothing about the sexual harassment allegations and I couldn’t tell him. I just turned to him and replied, “never say that again.”  I lost a friend that day. It hurt.  It was Charlie’s flirting with her that she objected to. She found him repulsive. At this point, I knew Alice well enough that I pointed out that I had said similar things (a phenomenally stupid thing to say), but she explained that she knew I didn’t mean anything. Charlie, in her eyes, was a creep.  In reality, Charlie was gay and had no interest in Alice, but he still liked to joke around. Of course, that in no way excused his behavior any more than I had an excuse for my behavior. Alice didn’t know Charlie was gay, but as a devout, conservative Christian, it certainly wouldn’t have made the situation better.  Eventually, Alice left the company (I don’t know the circumstances), and I learned a hard lesson about my personal behavior and what is and is not appropriate in the work place. I also began to get an understanding of how horribly women are often treated and realized that, even if she liked me, I had a hand in that treatment. It’s not a fun lesson to learn about oneself, but the most painful lessons are often the best.  My flirting played a role in my landing that job, and now it played a huge role in several people being hurt.  Leaving  I started going back to college to take COBOL classes and after six months, leveraged that knowledge and my limited experience to get a full-time job at an insurance company. At the time, it was the best paying job in my life. I was a finally a professional software developer.  My first programming job is still my “best” worst job because it was the platform that launched my career, but it was a nightmare. Good people, bad people, and cronyism galore. And decades later, I still feel bad about my role in the mess.  ","title":"How I Landed the Worst Best Job","url":"\/blog\/how-i-landed-the-worst-best-job.html"},{"body":"    An unusual event for Blue Origin Source    With the publication of the open letter to Jeff Bezos from his employees  , Jeff Bezos is continuing to have a Terrible, Horrible, No Good, Very Bad Year. Despite the hopes of many in the space community to see rich competition amongst private space industries (since the government ones are moribund), Blue Origin’s completely inability to do, well, anything, has dashed those hopes. SpaceX is clearly the winner and all Elon Musk had to do was to uncharacteristically say nothing.  And Bezos' woes are self-inflicted.  Note: All of the following is grossly over-simplified.  I often talk about SpaceX and Blue Origin with friends and many are surprised to find out that Jeff Bezos founded Blue Origin in 2000  , two years before Musk created SpaceX. Bezos has been pouring a billion dollars a year into Blue Origin while SpaceX came close to bankruptcy several times in its early years. Blue Origin follows a traditional, slow process of incremental advances, while SpaceX is applying agile project management to rockets . Insane!  And SpaceX is destroying Blue Origin. Just this year, SpaceX has had more flights than Blue Origin has had in over 20 years. And I’m being generous by counting Blue Origin’s test flights. Further, SpaceX is far cheaper than all of its competitors and it looks like its rockets are just as reliable, if not more so, than most of its competitors.  So where did Bezos go wrong?  Blue Origin has been rather quiet about their actions, so many things in the early years are speculative, but they did manage to get a small rocket named Goddard to fly to 85 meters in 2006   . And for the next few years, that was kind of it. This is probably a contributing factor to Blue Origin president Rob Myerson leaving the company in 2017  . Bob Smith of Honeywell was brought on board and, well, we’ll get back to him.  In 2006, the United Launch Alliance (ULA), was formed as a joint venture between Lockheed Martin Space and Boeing Defense. Prior to this, there was an a hope of increased competition between competitors that would lower the costs of space launches. Instead, Lockheed Martin Space and Boeing found a way to sidestep competition.    ULA’s Vulcan Rocket: Some assembly (still) required. Source    Later, in 2014, facing the prospect of the upstart SpaceX starting to cut into their market share, ULA decided to gamble and buy BE-4 rocket engines from Blue Origin. Was this to prop up a competitor? Was this to have access to cheaper rockets? Did they think they couldn’t do this themselves? Who knows what their reasoning was, but their gamble failed. The BE-4 engines were to be ready by 2017. The US Department of Defense was looking forwad to ULA’s Vulcan rocket launching with US-made BE-4s instead of the Atlas V rockets launching with Russian-made RD-180 engines.  As of 2021, ULA is still waiting for the BE-4 engines. Of course, it doesn’t help that, with limited experience, Blue Origin agreed to make some of the most difficult rocket engines you can make. The SpaceX initial rockets would never have passed DoD muster.  So, not many flights. Can’t deliver engines. What would you do in Bezos' position?  Well—and this is admittedly before the BE-4 debacle—in late 2011 when Blue Origin was struggling to deliver anything and SpaceX was picking up NASA contracts (thanks, in large part, to the hard work of Gwynne Shotwell), Bezos decided that the only reasonable thing to do was to start working on the New Glenn rocket  . But not a modest version as was earlier planned, but a huge version to rival the Saturn V in height. Suddenly, Blue Origin’s engineers were tasked with creating one of the biggest rockets of all time. Oh, it had to work perfectly the first time. There was no learning curve for Blue Origin. There was a learning cliff and Blue Origin lemmings were being herded over it by Bezos.  Call it a hunch. But maybe, maybe, Blue Origin should have learned to finish something before starting on such an astronomical goal.  So clearly Rob Myerson had to leave, but who’s Bob Smith?  Bob Smith was the man that Bezos picked to run Blue Origin. He has an impressive resume  , including a doctorate in aerospace engineering, a business degree from MIT, and many years of experience building rockets. He should be perfect.  Instead, it’s overwhelmingly agreed by everyone outside of Blue Origin that Smith has been a micromanaging disaster. On Glassdoor, 91% of SpaceX employees approve of the CEO  . For Blue Origin, only 22% approve of the CEO  . For most companies burning cash instead of rocket fuel, having a CEO in place for four years with little to show except demoralized employees and an insanely high staff turnover rate  , the CEO would be out the door.  This Ars Technica article sums things up nicely  :   As Blue Origin has fallen behind SpaceX in recent years, morale has declined, according to multiple sources. Bezos pushed the company further from its roots in late 2017 by sidelining long-time president Rob Meyerson and hiring a traditional aerospace veteran, Bob Smith, to become chief executive. Coming from Honeywell, Smith instituted a more bureaucratic management style, and Blue Origin’s progress seemed to slow significantly. Whereas Bezos' debut flight on New Shepard could have occurred as early as 2019, it slipped by months and eventually years. Critics of Smith’s plodding management style started referring to the company as “Blue Honeywell.”   Jeff Bezos is going to fire Bob Smith. We don’t know when. The only question is why he hasn’t done it yet.  It’s almost as if even Bezos doesn’t believe in Blue Origin’s mission.  I want Blue Origin to succeed. I want Jeff Bezos' company to be one of the most wildly successful companies in history, alonside SpaceX, Rocket Lab, and any other company that’s try to push humanity into space. We need competition here. But where is it? ULA is hanging on with government contracts. Roscosmos wants out of the ISS because they’re so strapped for cash. The ESA and EU are trying to force EU countries to only buy from Arianespace, but the cracks are beginning to show  . Honestly, Blue Origin could have been the one to watch in this space. And now we’re watching them, but not for the reasons they wanted.    Symbolism.   Ironically, part of the issue seems to be that Blue Origin employees know that Bezos won’t stop spending. The money won’t stop arriving. They discuss this in articles, in the Glassdoor ratings, and common sense reinforces this: SpaceX was lean and hungry, Blue Origin has never been either. Like many startups who are flush with cash and fail because they can’t keep their eye on the ball, Blue Origin has no reason to push hard.  And now, Blue Origin’s lawsuit against NASA is destroying their last opportunity to succeed  . NASA, uncharacteristically, has slammed Blue Origin. If you’re unfamiliar with the lawsuit, NASA solicited bids for their 2024 moon lander. Blue Origin’s bid was ridiculously over NASA’s budget and NASA rejected it. So while SpaceX at one point sued the government for the right to compete, Blue Origin is suing because they couldn’t compete.  Here are a couple of tidbits from that article:   Blue Origin was “able and willing” to offer NASA a lower price for its lunar lander but chose not to because it expected NASA to ask and negotiate for a lower price first, the attorneys allege, citing a six-page declaration written by the company’s senior vice president Brent Sherwood in April.   Given that Blue Origin has not demonstrated a track record of being able to repeatedly fly to and from space, why on earth (heh) would Blue Origin think they were a front-runner in the bid or that NASA would come back to negotiate? When NASA, unsuprisingly, said “no”, Blue Origin went to the GAO (Government Accountability Office) and complained it was unfair. The GAO also told Blue Origin “no.” So Blue Origin sued.  Worse, because the lawsuit has halted much of the moon landing program, NASA lawyers wrote the following:   “What begins as a mere procurement delay all too easily turns into a lack of political support, a budget siphoned off for other efforts, and ultimately, a shelved mission.”   We’ve been waiting fifty years to get back to the moon. Blue Origin decided that their pride was more important than the dreams of the American people. It’s not like the fate of their company hangs on this: Bezos won’t stop spending, but they’ve dug themselves a deep hole, lost the goodwill of the American people, and as NASA insiders have said, they’re unlikely to get any government contracts after this stunt.  Seriously, Mr. Bezos, if you believe in your stories about space industry, its time to fire Bob Smith and bring in someone who can fix this damned mess. Everyone already knows he’s gone. Let’s get this done and give Blue Origin a chance to compete.  ","title":"What's Wrong with Blue Origin?","url":"\/blog\/jeff-bezos-1000-problems.html"},{"body":"   Background : I recently proposed a new object system, name Cor, for the Perl core  . This work has been done in coordination with Sawyer X and Stevan Little. What follows are my musings on the process and OO in general. For those things I inevitably get wrong in this discussion, it’s my fault, not theirs.   Contrary to what some might think, the discussions about the Cor OO system are ongoing, but it’s happening quietly, via email, and the holiday period doesn’t help.  And the Moose   has function also doesn’t help. At all. It’s possible that you’ve heard some rumblings about the has function, so let’s clear up some of this. This will take some time, but I promise you, by the time I’m done, that I will have given a rubbish explanation of a hard problem. In throwing this together, I truly understand Blaise Pascal’s comment “I would have written a shorter letter, but I did not have the time.” If I had the time, this would be much shorter and clearer, but I’m too lazy to delete this.  In Moose (and Moo  , but we’ll stick to Moose for this discussion), the has function lets you declare an attribute:  package Point2D { use Moose; has [qw\/x y\/] =&gt; ( is =&gt; 'ro', isa =&gt; 'Num', required =&gt; 1, ); }  The above is a class for the canonical 2D point object that developers have a love\/hate relationship about when learning OO concepts.  As you can see from the has declaration, we have:   Created slots for the data  Created read-only accessors for the data  Required that they be passed to the constructor     That’s a lot to pack into one function, and it leads to confusion like this:  has attr =&gt; ( is =&gt; 'ro', # read-only isa =&gt; 'Int', # declare it an integer writer =&gt; 'set_attr', # but it's read-only? required =&gt; 1, # 'attr' must be provided builder =&gt; '_build_attr', # called if not passed in the constructor );  There are other examples you could put together, but that shows part of the issue: Moose has crammed so much into one function that it’s easy to write code that is confusing or just plain doesn’t make sense. The above does make sense if you realize that required doesn’t mean that the attribute is required to be passed to the constructor (as some assume). The docs say this  :   Basically, all [required] says is that this attribute must be provided to the constructor or it must have either a default or a builder. It does not say anything about its value, so it could be undef.   But what does declaring something as read-only and providing a writer (mutator) mean? That’s confusing because we’re really packing a lot of meaning into a single function.  What’s worse, it’s harder to write code that does what we need. If I want a private, internal slot, per instance, with no accessor, do you know how to do that in Moose? We tend not to do that too much with Moose simply because it’s hard to do, despite being trivial in other languages.  Well, you can declare a slot as bare (instead of ro and friends) and then do:  my $value = $self-&gt;meta -&gt;get_attribute($attribute_name) -&gt;get_value($self);  Except no one does that (by “no one”, of course, I mean “the set of no developers who aren’t not Stevan.“) They just declare the attribute private and that’s that. And I don’t blame them. But it means that everyone just makes accessors for everything, and that makes it much easier to make public accessors for everything and when I’m cleaning up code for a client, that’s a problem because it makes it much harder for me because now I have a public contract that I have to respect, even if there wasn’t originally a need to expose that data.  Encouraging people to write accessors is a bad idea, but people do it all over the place, just as they often make attribute mutable for no good reason ( and here is a great example of why mutable objects are dangerous  ). A good rule of thumb for OO design: make your interface as small as possible. Moose, unfortunately, offers an affordance to making our interace as large as possible.  Moving along, what does that point class look like in Java?  public class Point2D { \/\/ slots private double x; private double y; \/\/ accessors public double x() { return x; } public double y() { return y; } \/\/ constructor public Point2D(double x, double y) { this.x = x; this.y = y; } }  Note how the slots, the accessors, and the constructor arguments are all nicely decoupled, leaving no ambiguity.  And without going into detail, private slots with no accessors are trivial in Java, at both the class and instance level.  But where does that leave Cor ? Well, Cor aims to be core , meaning that overloading has is problematic. We’ve learned a lot from Moose about how to make OO that feels “perlish”, but we need to keep growing and learn from our mistakes. Returning to this:  has attr =&gt; ( is =&gt; 'ro', isa =&gt; 'Int', writer =&gt; 'set_attr', builder =&gt; '_build_attr', required =&gt; 1, );  For Moose, that’s OK. For Cor , that’s not, because it means you can write code that is, at best, confusing.  And then there are things like this:  my $auth = Authenticate-&gt;new($one_time_token); # or my $auth = Authenticate-&gt;new( user =&gt; $user, pass =&gt; $pass ); # or my $auth = Authenticate-&gt;new({user =&gt; $user, pass =&gt; $pass});  Here’s one way you could try to handle that:  around BUILDARGS =&gt; sub ($orig, $class, @args) { my %arg_for = @args &gt; 1 ? @args : 'HASH' eq ref $args[0] ? $args[0]-&gt;%* : ( token =&gt; $args[0] ); return $class-&gt;$orig(%arg_for); };  In languages with multi-dispatch, if we add a new way of instantiating an object, we can just add a new constructor and the language will handle the dispatching for us. With Moose, we need to use BUILDARGS for this, and manually handle all of that dispatching ourselves. That means more bugs. If Perl had something like multidispatch, we could possibly write:  method BUILDARGS (Code $orig, Class $class, UUID $token) { ... } method BUILDARGS (Code $orig, Class $class, Str :$user, Str :$pass ) { ... } method BUILDARGS (Code $orig, Class $class, HashRef $args) { ... }  For Cor , we’re not going to get there because Perl doesn’t yet have robust signatures like that and we’re probably never going to get multidispatch, but decoupling the very overloaded meaning of has will help, and limiting how we can pass arguments can help (no more “named pairs” or “hashref” decision making).  And here’s a fun one! An attribute that you can’t set, even though it looks like you can pass it in the constructor:  #!\/usr\/bin\/env perl package SomeClass { use Moose; has 'username' =&gt; ( is =&gt; 'ro', isa =&gt; 'Str', required =&gt; 1, init_arg =&gt; undef, default =&gt; 'Bob', ); } # this prints &quot;Bob&quot; print SomeClass-&gt;new( username =&gt; 'foo' )-&gt;username;  Of course, Cor could try to trap every possible illegal (or confusing) combination and then what?   Warn about them?  Ignore them?  Make them fatal?  Do this at compile-time?  At runtime?   And if we check illegal combinations, then if we ever extend has to have more functionality, we have to figure out the new “illegal” combinations (such as making attributes simultaneously required and lazy).  Do we stop developers from shooting themselves in the foot or hand them a gun?  The counter-argument I hear from many developers is “we don’t need to separate slots and attributes or a different syntax for declaring attributes. What we’ve been doing works.”  And frankly, this has worked fairly well. If the new Cor proposal doesn’t provide some dead-simple way to create accessors, then we’ll wind up with 42 different modules to provide that and we won’t be that much further along than when we started.  There are so many decisions to be made with Cor , many of which would be dead wrong if we move too quickly and we don’t want to get this wrong. And the lack of multi-methods and the inability to introspect signatures means the BUILDARGS and BUILD pain will remain in some form (though BUILD isnt' too bad).  Thus, we really want to have some separation of slots, attributes, and constructors, but making an easy syntax for that is is a challenge. As the old line goes, “making things hard is easy; making things easy is hard.” And making a solution that is easy and does the right thing and that all Perl developers will like is impossible. And there will be developers looking at the final solution and sayind “no” because their one pet peeve wasn’t respected.  As for our initial work with the syntax, the core Perl devs liked what they saw in the initial proposal, but it’s one thing to say “shiny!“. It’s another thing to hammer that shiny onto something that already exists. There are a number of approaches which can be taken, but they require hard decisions. For now, Cor will be focusing on the syntax.  ","title":"Moose \"has\" a Problem","url":"\/blog\/moose-has-a-problem.html"},{"body":"    A Brief History of Private Spaceflight  The Four Phases of Market Evolution  Spaceflight is Entering the Stability Phase  Starship Changes Everything  Costs  Disruption  What the Future Holds  New Industries  Mining in Space  Disruptive Uses?  Investing  Be Patient     A Brief History of Private Spaceflight  At the end of the 20th century, spaceflight was considered a “national” endeavor, limited to a few countries such as the United States, Russia, France, China, Japan, and so on. Companies like Boeing, Honeywell, Roscomos, Arianespace, and others provided space offerings that were attached to national programs, not private programs.  But at the dawn of the 21st century, things started to change. Jeff Bezos founded Blue Origin  in 2000. Elon Musk founded SpaceX  in 2002. Richard Branson founded Virgin Galactic  in 2004. Blue Origin was in stealth mode. Elon Musk was laughed at. Virgin Galactic was viewed as the toy of a billionaire playboy. Bezos, Musk, and Branson, serial entrepreneurs all, saw a future for spaceflight that others had not. They wanted to create new industries.  At that time, VCs weren’t interested in space startups. Investment funds didn’t carry them in their portfolios. By 2014, when Google acquired satellite company Skybox Imaging  , that changed. The era of private spaceflight was starting to mature.    The OTRAG Rocket, 1977   But just as many people aren’t aware that the internet started long before the mid-90s, private spaceflight began long before this century.  Commercial satellite launches began in the 1960s, long before humanity landed on the moon. The first amateur satellite, OSCAR I, was launched in 1961  , a scant four years after Sputnik, and the first commercial satellite, Telstar 1, was launched in 1962  .  And then there was OTRAG. OTRAG was a private rocket company founded in 1975 by Lutz Kayser, a German entrepreneur. The intent was to build low-cost rockets an order of magnitude cheaper than existing rockets. Amazingly, Kayser succeeded, but the history is tragic  .   Kayser’s activities made the great powers nervous. The USSR and France were not interested in Germany achieving an indigenous long-range rocket activity. American rocket makers were not interested in having a low-cost competitor. A propaganda campaign began, alleging OTRAG was a cover for German and South African nuclear cruise missile development. Crude Soviet-source disinformation was eagerly picked up and given credibility by the American mainstream media. The government of the Congo, [where OTRAG was launching from], was pressured by the Russians to withdraw permission to use the site. OTRAG left the country in April 1979.   They relocated to Libya, but the Libyan government seized their rockets in an attempt to create their own rocket program. Fortunately, without Kayser’s experience, they failed, and the dream of private spaceflight failed with it.  If Kayser succeeded, we might have our moon base by now. We might be on Mars already. Instead, we’re still waiting, decades later.  There’s a decent Wikipedia page that sums up the history of private spaceflight  , but mostly, the nascent industry sputtered along in fits and starts, usually followed by bankruptcy.  It wasn’t until the 21st century that the private rocket industry exploded.    Now, everything is changing, but what’s going to change? To understand that, we have to know a bit about the evolution of markets.  The Four Phases of Market Evolution  Note : much of the following is derived from the work of Simon Wardley  , but I’ve adjusted some of the terminology.  Markets (and new products within them) tend to evolve through four maturity phases. Each has unique characteristics and should be treated differently. This is boring for most, but it’s important. These phases are:   Novelty Phase : A new market emerges. Unknown how\/if consumers will respond.  Stability Phase : The new market is stabilizing, but not necessarily mature. Still experimenting with different approaches.  Commodity Phase : Robust competition amongst competitors. Less product differentiation due to market maturity.  Utility Phase : Market is mature, widespread, very stable, and margins are thin. New markets are built on top of these.    Note : the above descriptions are far too brief. For extended descriptions, I have a brief article which explains them in more detail, including examples .   For private spaceflight, the “novelty phase” was when entrepreneurs started to see potential, but it’s very high-risk, high-reward. As first movers, if they succeed (most did not, of course), they can reap tremendous benefits. This is why Elon Musk regularly tops the lists of the world’s richest people.  As we move from novelty to stability to commodity to utility, we see risk gradually decreasing:    Novel markets are very high-risk.   Why does risk decrease? There are various factors, including customers becoming more comfortable with the market and having new ideas how they can utilize the market. But to succeed in this evolution, the products in that market must be reliable . Would you buy a cell phone that only has service 50% of the time? Probably not.  What this means is that as markets mature, their products become more stable:    Stable markets are very predictable.   Now compare this with air travel  :   In the industry’s embryonic era, accidents were disturbingly numerous. Ironically, 1929, the year of “The Great Crash,” was also one of the most crash-ridden in aviation history, with 24 fatal accidents reported. In both 1928 and 1929, the overall accident rate was about one per every million miles flown. In today’s system, an accident rate of that magnitude would result in a nearly incomprehensible 7,000 fatal accidents each year.   By 2017, there were no recorded deaths of passengers in commercial passenger jets  . In 2019, a National Safety Council’s recent odds-of-dying data  reports that you have a 1 in 107 chance of dying in a car crash. Airline passenger? “Too few deaths in 2019 to calculate odds.”  But what about spaceships? Astronaut Chris Hadfield reports that in 1981, you had a 1 in 9 chance of dying in a space shuttle launch. By 1995, the odds of dying had already dropped to 1 in 38. Of course, those are per flight, not per lifetime, but if this trend continues, will we see ourselves in the bizarre position of seeing our odds of dying on a spaceflight lower than our odds of dying in a car crash?     Today, no astronauts have died since the 2003 Columbia space shuttle disaster  . (Note: the last cosmonauts to die in space were in 1971  ). Though in one bizarre incident, an astronaut almost drowned on a July, 2013 spacewalk  .  That doesn’t mean spaceflight is suddenly safe, but it’s far safer than it ever has been.  Spaceflight is Entering the Stability Phase  Long viewed as a novelty, spaceflight is entering the stability phase. It’s not there yet, but it’s on the verge and SpaceX is leading the way. While the novelty phase offers higher rewards, its risk level is too high for all but the most informed (and wealthy) investor. However, when the stability phase is entered, the market has begun to solidify, the products are becoming trusted, more investors can understand the market, and the risk has dropped tremendously. Of course, the reward has dropped, too, but that seems a fair trade-off.    Advanced Blue Origin Lander Prototype.   However, particularly if we’re viewing things through a capitalist lens, it’s hard to argue that a market is “stabilizing” if there’s only one real supplier. Blue Origin, despite their recent suborbital tourist flights, is struggling . Due to poor performance, investors worry about Virgin Galactic  and the new lawsuit against Richard Branson and Virgin Galactic chairman Chamath Palihapitiya over allegedly dumping shares while covering up spaceship defects  is not going to help matters.  Roscomos, the state-owned Russian space company has long struggled financially and recently announced steep budget cuts  are not going to help. Private Russian spaceflight company Dauria is fighting bankruptcy  . French company Arianespace reported a 30% increase in 2021 revenue  but their lack of reusable rockets and their promised reusable “Prometheus” engine (to be first launched eight years from when I’m writing) being delayed means they’re falling further and further behind SpaceX. Worse, the impending arrival of Starship means Arianespace is in serious trouble  .  So with them not going anywhere, what happens if SpaceX collapses? Elon Musk has already warned his employees about potential bankruptcy if it can’t speed up production  (though he later admitted this was unlikely). But I’ve talked to plenty of people about the space industry and most of them are unaware of the numerous successful competitors out there.  Virgin One was spun off from Virgin Galactic and shortly after its first successful launch of seven satellites  announced that NASA put them on their short list of satellite launch providers  . This doesn’t guarantee they’ll be selected for launches, but the private sector is watching them. They can’t launch very large satellites, but their launches are only $12 to $15 million per launch  . They’re a force to watch.  Rocket Lab and Firefly are also looking very competitive  :   Rocket Lab’s small Electron rocket goes for about $7 million per launch, Firefly’s medium-sized Alpha rocket goes for $15 million per launch, and SpaceX’s Falcon 9 is advertised at $62 million per launch.   There’s also UP Aerospace  , United Launch Alliance  , SpaceForest  (services starting in 2023), and many others starting development. Raphael Roettgen  , on the TerraWatch podcost  , estimated that there are around 200 launch startups around the world today.  The market is booming and stabilizing. Customers can now choose between a wide varieties of rockets with different payload sizes and price points, but there’s still an enormous amount of risk and it will be a couple of decades before we can approach the commodity stage.  Starship Changes Everything  A few years ago, a friend in Nottingham told me she wanted to spend the weekend in Ireland, so she bought a plane ticket. When she changed her mind, she threw away the ticket. It was only £20. No longer the domain of the wealthy, air travel is for everyone and this is (eventually) coming to space.  Costs  For the space shuttle, the average cost per launch for the last decade was around one billion dollars  . For Starship, Musk expects launch prices of less than ten million within two to three years  . Eventually, he projects those prices could be less than a million dollars a launch.    In contrast, the Concorde’s inaugural 1973 flight from Paris to Rio de Janeiro  , roughly 5,700 miles (9173 km), operating at a cost per flight of up to $5.20 per mile  , could be looking at just shy of $30,000 for that flight. Adjusted for inflation, that’s almost $190,000 today. However, the $5.20 figure was for a 1972 estimate. In reality, the Concorde was subject to wild cost overruns, so a $190,000 flight cost is certainly low.    That Starship may not be significantly more expensive than a Concorde is mind-boggling. The fact that Starship might actually cheaper than planes for long-haul flights is even more mind-boggling  .  But wait, there’s more! The cost of building the Boeing 737  , one of their least expensive aircraft, is just over $100 million. At the high end, Starships could cost around $37 million each  , but an optimistic mass production rate could drop that price to $5 million per Starship. This is insane .  Disruption  The lower costs, however, have a knock-on effect for the entire industry. The media tends to think of the cost of the launch, but the insanely high launch costs ripple through the industry. Want to make a lunar lander out of stainless steel? Too bad. Titanium is about 35 times as expensive as stainless steel  , but it’s weighs half as much. With costs per kilogram in the thousands  , every penny counts.       Launch Vehicle  Payload cost per kg      Vanguard  $1,000,000    Space Shuttle  $54,500    Electron  $19,039    Ariane 5G  $9,167    Long March 3B  $4,412    Proton  $4,320    Falcon 9  $2,720    Falcon Heavy  $1,400    Starship  $100      Yes, Starship costs can potentially cost an order of magnitude less per kilogram. What else does that mean?  It means missions can be run in parallel, not in series.  Think about a string of electric lights. If they’re run in series and one goes out, they’re all out. If they’re run in parallel and one goes out, the rest keep working and you can compensate for the missing bulb.  Missions in space today are long series of single points of failure once launched. Consider a simple (ha!) robotic rover to Mars. It’s been devilishly hard to land on Mars, so we have to:   Design the rover  Order the parts for the rover  Ensure all tolerances are perfect for the rover  Assemble the rover  Test, test, test, and retest the rover  Return to step 2 if the parts aren’t up to snuff  Return to step 1 if the design isn’t up to snuff  Do all of this in a near-sterile environment so we don’t contaminate Mars  Move the rover to the rocket  Have a perfect launch  Wait six to nine months while the rocket heads to Mars  Enter the Martian atmosphere at just the perfect angle  Survive the descent to the surface  Scream in agony as the dust storm covers the solar panels and kills the mission   I skipped a few steps in there, but prior to launch, costs are astronomically high to due having to be perfect , absolutely perfect , because once you launch, any failure at any single point renders all of the previous work moot.  With Starship, you could build multiple rovers, with much less costly materials (weight no longer being the major consideration), lower tolerances for many of the components, and fly all of them to Mars on multiple launch for less than the cost of a single rover today. A couple of flights fail? Fine, you still have a fleet of rovers on Mars. And no, you can’t reduce all of the tolerances, but by building a fleet, the individual costs can be lowered as you gain more experience in procurement and assembly.  Heck, the Perseverance rover to Mars will have a projected final mission cost of almost three billion dollars  . Musk’s long-term goal is to make a passenger ticket to Mars as low as $100,000  .  Starship changes everything.  What the Future Holds  Just as the internet has proven to be a hugely disruptive force for industry and society, so will the new space industry. But like the internet, we don’t know what that impact will be.     New Industries  As we improve our rocket technologies, many of the new industries are already on the horizon. Space tourism on rockets and even the ISS have already happened. But space hotels are being actively planned  :    Orbital Assembly Corporation space hotel concept.   Real-time satellite imaging is widely available, high-speed, low-latency internet  is starting, and Japanese company Astroscale  is already planning to remove space debris and provide on site inspection and maintenance of satellites. ( Dealing with the ever-increasing threat of space junk is a serious problem  ).  Mining in Space  Of course, no story would be complete without discussing mining. If you are at all interested in space, you’ve long heard that the world’s first trillionaires will be space miners. The early space mining companies are gone  because they, like OTRAG, were early adopters who adopted too early. They generally anticipated that the first space mining would involve mining extremely valuable ... water.  Yes, water. They were going to sell it to spacecraft using various means of water propulsion  (seriously, the idea is picking up steam). However, the industry they wanted to support wasn’t ready to be born, so the space mining companies drowned (forgive me). But water is going to be a seriously valuable resource in space. I describe this briefly in one of my public presentations  (if the video doesn’t start at the correct spot, jump to 50:13):      Side note: I’m also a public speaker, so if you want to have me give a talk about this, let me know !   But what else might we mine? We are facing a near-term global lithium shortage, so that would be awesome. But where do we find it? Lithium dissolves easily in water and is found in salt flat brines like those of Chile, or pegamites , a type of rock that forms in water-rich magmas. It’s estimated that the moon may have the same relative amount of lithium as Earth, but without water, it wouldn’t be concentrated enough to efficiently mine. Mars? Maybe, but we’re not sure. Ceres has been speculated as a source of minable lithium, but we can’t send our mining robots on a hunch. We need to go prospecting to find what we need and currently, that’s too expensive—until Starship is fully operational.  The same goes for Rhodium and the other platinum-group metals. Without a low-cost ability to to prospect space, we can’t send out mining equipment, much less miners.  But let’s say we find a near-Earth asteroid with the metals we want. The cost, per load, of those materials needs to be lower than the cost of building our prospectors, launching, searching, building our mining robots, launching them, mining the metals, refining the metals on site, and then returning them to Earth. As it stands now, we can’t meet those costs and even with Starship, getting them back to Earth is difficult. We can’t use parachutes. Simply crashing them on our planet will destroy much of it in the atmosphere and needlessly endanger lives, so they need to be brought back in a rocket. It’s simply not economically feasible.  More accurately, asteroid mining isn’t economically feasible until we have an established industry in space . At which point, all of that changes because we can build, launch, and return in space. That’s much less expensive and we can use those materials to build out space-industry in a more cost-effective manner.  Sadly, that’s decades away. But asteroid mining might still be viable.  The moon is regularly hit with asteroids and probably about 20% of those are metallic. Building stations on the moon and using the materials there might be worthwhile...and we’re already looking at lunar stations. Thus, we can do asteroid mining, so long as we’re doing it on the moon.  The south pole of the moon is a potential site for setting up our stations  . It appears to have abundant water and if we send out plenty of robotic prospectors, finding a location near silicon and platinum deposits would give us the ability to mine the materials to make solar panels to power our lunar outposts. But we won’t know until we start prospecting.  Disruptive Uses?  We’ve already discussed Starlink  , a mega-constellation of satellites to bring high-speed, low-latency internet connections to the entire planet. There are considerable challenges involved, including efforts to reduce their visibility  and not ruin ground-based astronomy. The long-term goal is to have over 30,000 satellites operational, so effort is being put into ensuring defunct satellites de-orbit as quickly as possible to mitigate the problem with space debris  .  But it’s possible that Musk may be planning to become a trillionaire by creating space-based solar power generating stations  . These generate nice, clean solar power in space and beam it back to Earth. Before you think this is just science fiction, China is already planning on doing this  , but unlike SpaceX, they don’t have access to Starship. That being said, the the long-term benefits outweigh the costs of launching via China’s Long March 5  rocket. China is pushing forward.  Musk’s path may be less clear because if successful, he threatens the fossil-fuel industry and they will do everything they can to stop him (unless he creates a partnership with them, which I doubt). They will be lobbying like made, citing environmental concerns of the launches (oh, the irony), trying to find regulatory means of stopping him, and pointing out to politicians all the jobs their districts stand to lose if these stations are operational. This seems strange, but we’ve already seen this with Tesla Motors. This will be OTRAG all over again: political and market forces working hand-in-hand to block innovation.  There will be other disruptive markets to emerge from this industry sometimes being referred to as NewSpace  , but like Black Swan events  , we can’t predict them in advance, and they will have enormous impacts on markets.   Note : space elevators will not be the disruptive force  . Not only are they probably decades away, but an ascent to space will likely take weeks.   Investing  Like OTRAG, early space mining companies died because they were too soon. Early investors in this “novelty” stage of the market lost a lot of money. Now that we’re entering the stability phase, investing presents less risk, though less reward.  Before NewSpace started entering the stability phase, there were few, if any, VCs or financial services company offering services. Today, there are many of them. Today, investment giant Morgan Stanley is even offering space investing  . However, space investing today is mainly in the hands of the wealthy. Sadly, this is probably a good thing because as if often pointed out by investors, unless you put in 10,000 hours or more in deep study of NewSpace, you won’t have the expertise to evaluate investments.  NewSpace can’t do the early internet thing of “throw money at the wall and hope it sticks.” The amounts of money involved means you have to have an extremely high chance of a payoff. These companies aren’t investing in new technologies so much as they’re investing in markets with solid business plans. This sets this industry apart from the early days of the tulipmania  Internet Bubble.  Of course, other companies are now getting involved. The highly-regarded International Space University  (ISU), has a Master’s Programs in Space Studies and offers a wide variety of professional courses in different aspects of the space industry.  Be Patient  We’re at the NewSpace equivalent of the internet in the mid-90s. It’s booming tremendously, but due to the extremely high costs and high risks, it’s being approached in a healthier manner than the internet boom. However, we won’t see internet speeds of advancement. For much of the internet, it was a matter of building ISPs and connecting people and letting the industry blossom. For NewSpace, the entire industry needs to be boostrapped, including the core infrastructure.  It’s not going to impact us directly today–no one is buying a cheap flight to an orbital hotel–but it’s coming and despite the naysayers, it’s not stopping. And even if SpaceX unexpectedly goes bankrupt (it won’t), it will still keep coming. I don’t know if I’ll live long enough for a vacation in space, but my daughter will. So will your children. So will many of you who read this.  The long-awaited promise of a future in space is finally arriving.  ","title":"Time to Invest in Space?","url":"\/blog\/time-to-invest-in-space.html"},{"body":"  In 2018, the documentary Amazing Grace  featuring Aretha Franklin was first released. It was filmed in 1972. You might wonder why it took almost five decades for a documentary of the top-selling gospel R&amp;B album of all time to be released. Sure, Franklin sued twice to prevent its release, but we’ll skip that. The real reason is hilarious.     Sydney Pollack  , now known as one of the most influential filmmakers in history, was approached by Warner Brothers to film Franklin’s concert in a church. He agreed immediately upon hearing her name. The camera crew set up and by the second night, word of her electrifying performance had spread so quickly that even Mick Jagger was in the audience. This was a legendary event.  Warner brothers didn’t want to wait five decades for the film to be released, but to understand why it took so long, we need to understand a bit about how a movie is made.  You know how you take 100 pictures and maybe one or two are any good? That’s a documentary. You see maybe an hour or two of something amazing, but behind the scenes (pun intended!) are many more hours of footage that never leave the cutting room floor. Some of that footage is awful. Some may be amazing, but it doesn’t fit the director’s vision. For whatever reason, you shoot tons of footage and use almost none of it. For Amazing Grace, twenty hours of footage became 87 minutes of film. In other words, less than 10% of the footage was used.  A film camera, as you know, records images, not sound. That’s for the microphone. The film and the sound for the film are thus recorded separately.  Later, when putting together a rough cut of the film, the “sync sound” (the actual sounds recorded while filming) gets synced with the film. You’ve probably seen films where the sound doesn’t quite match the actor’s lips. This is sometimes known as “lip flap” and it’s annoying as hell, so you need to get it right. With twenty hours of film and twenty hours of sound recording, how do you get it right?  Typically, we use a clapperboard  , also known as a “film slate.” There are various styles, but they usually include at least the date, the production, the scene and the take. But the one thing they always include is that long bar across the top. You’ve all seen video of someone yelling “lights, camera, action!” And then the second assistant camera  holding the slate slaps the top down with a sharp “clack.”    A film clapperboard Source    In the rough cut of the film, you’ll see that clapperboard being clapped. In the audio recording, there’s a sharp spike when that clapperboard makes that noise. You can then use those to sync the film and the sound.  Sydney Pollack forgot to bring clapperboards to the church. After, they discovered they couldn’t sync the sound to the film. They shot the world’s greatest gospel documentary as a silent film.  By June of that year, Aretha Franklin’s album had sold millions. The movie would have been a slam dunk. Via Hollywood Reporter  :   The choir director from the Watts recordings was brought in to try to lip-read the reels, but after months of work, only about 150 minutes of footage had been matched with sound, none of it adding up to a complete, useable song.   It wasn’t until 2008, using computers to synchronize the film and the sound, was the movie finally complete. Aretha Franklin sued (twice) to prevent the release of the film. After she passed, an agreement was reached with her estate to release the film. It grossed less than $6 million dollars, a far cry from what it could have been, but Metacritic rates it 94 out of 100  , or “Universal Acclaim.”  Of course, Aretha Franklin deserves nothing less.   If you’re interested in the film industry, Castaldo Academy  has produced a short video explaining the clapperboard’s use.     ","title":"The Documentary That Took 46 Years to Release","url":"\/blog\/the-documentary-that-took-46-years-to-release.html"},{"body":"    Introduction  Forget Europe  Working Holidays  Teaching English  Volunteer Work  Au Pair  Studying Abroad  Joining the Military  French Foreign Legion  Foreign Service Exam  Fantasy versus Reality  Conclusion     Introduction  Constantly on Reddit’s IWantOut community  , or if they’re a bit clued in, in expat forums, I see variations of the following question (the following was a real post on the now-defunct “Yahoo Answers” forums):   I would mainly like to move to Germany but I am wanting to move to Europe and learn fashion design. I don’t know what sort of job I can get over there being so young and not speaking another language. I don’t feel at home in the USA and finally did in Europe and I have to go back ! what should I do?   I call these fairy tale posts.  To figure out how to move to another country, you have to take stock of what you skills you possess which might help to make this happen. Here’s the typical profile of people making fairy tale posts:   No life experience  No work experience  No education  No language skills  No money   This does not bode well for your chances, but you might have the one skill necessary to pull this off: determination.  To start with, read Why you’ll say “no” to living abroad . I consider this the foundation post of the expat posts because it’s critical to understanding the process. You have to make sure that you can seize any opportunity which comes along (this doesn’t mean you should take any opportunity).  The very first thing you need to do is apply for your passport. If you don’t have a passport, you’re not serious about moving abroad. If you can’t be bothered to get a passport, you may as well stop reading.  The second thing you need to do is take stock of your life and stop doing anything which might land you in legal trouble. Of course, that’s a very personal decision, but if you wind up with a criminal record, your limited chances of living abroad are approaching zero. Maybe you don’t think the government should outlaw drugs, but your personal beliefs are meaningless when another country sees you have a drug conviction. You need to decide what’s more important to you.  The third thing you need to do is set your life up so you can leave as soon as is feasible. Long-term leases or owning pets might be an obstacle. However, working towards a college degree is a benefit, even if you don’t get to leave tomorrow. Always ask yourself, “what’s stopping me from leaving now?” As agonizing as this may sound, it also means considering your partner, if you have one. Would you leave a girlfriend\/boyfriend for a chance to move to New Zealand? You have to understand your motivations here and you should be prepared for tough choices.  If you do have serious work skills, and by “serious” I mean “highly skilled labor”, I’ll later be sharing a “How to get a work permit” series, but if you’re young and unskilled, you probably don’t have a chance at a work permit. So what do you have?  Forget Europe  I don’t really mean that, but if you think “where can I move” instead of “I want to move to Europe”, you’ll have many more opportunities. The biggest mistake I see many would-be expatriates make is forgetting that the world is round. For example, Montevideo is a large, cosmopolitan city and Uruguay is relatively easy to emigrate to, basically requiring around $1,500 a month income and a lot of paperwork  . Can you find a remote job that pays more than $1,500 a month? Maybe you can get to Uruguay. However, one person who could have done this told me “I don’t want to live in the sticks” (US slang for “middle of nowhere”). Montevideo is not “the sticks”, but if you don’t do your research, how would you know?  Working Holidays  For young people, you might want to consider working holiday schemes  . Australia and New Zealand offer them and while they aren’t designed to allow you to permanently live there, once you’re there, your options will open up tremendously.  Teaching English  Many young people complain that they don’t want to teach English abroad  . Too bad. It’s one of the most popular ways that young, unskilled workers can move to another country. Many people report heading to countries with a strong demand for English teachers and finding work under the table, but this is very risky. You might not find work and, if you do, you’re probably an illegal immigrant and risk deportation. Being deported from a country means you probably can’t get back in for many years. In fact, some countries won’t let you in if they find out that you’ve been deported from another country.  Volunteer Work  Many people are surprised to find out that volunteer positions around the world often require you to pay for the privilege of being a volunteer. Sadly, this is not a scam. Many volunteer organizations are poor and simply can’t afford to fly you to Paris, put you up in a flat and feed you while you work for two hours a day baby sitting someone. Do your research as some “volunteer” organizations are scams, but if you find a reputable volunteer organization, you can at least have a short time in another country and gain valuable life and work skills. Plus, it looks great on your résumé\/CV.  Au Pair  An au pair is basically a young person (usually a woman) who travels to a new country, stays with a host family and looks after the children for a time. It’s usually a short-stay trip and an au pair is not a nanny. Families do this to expose their children to other cultures and, sometimes, because they enjoy giving people from other countries an opportunity to see the world. Be careful, though. While it’s rare, some au pairs find themselves horribly abused by their host families. Work through a reputable agency and be safe.  Studying Abroad  Studying abroad offers you not only the chance to live in another country, it allows you to earn a degree which makes staying abroad much easier. In fact, many countries give preferential immigration treatment to students who have graduated from that country’s universities.  Many foreign universities are eligible for US federal financial aid. Other countries, such as Germany and Norway, have free university programs which foreigners are eligible for. This probably means staying in the US for a couple of years, going to university and earning good enough grades to qualify for foreign universities. If you can do this, it’s one of the best ways of getting out and staying out.  Joining the Military  This one is controversial, but yes, the military will often send you abroad. It’s one of the easiest ways of gaining life skills, work experience, seeing the world and saving money for college. Of course, you might have to kill the foreigners you meet. Your own values\/political beliefs will have to be examined strongly before you consider this.  French Foreign Legion  Sorry ladies, this one is for the men only.  I used to run a very popular (now defunct) expat blog and every week, searches for “French Foreign Legion” showed up in the list of most popular searches on the site. I did quite a bit of reading about them and learned that many people think the Foreign Legion of today is the Foreign Legion they saw in black-and-white movies, or read about in Soldier of Fortune magazine. In particular, there are three myths I want to dispel about the modern Legion:   They no longer accept felons  The no longer grant French citizenship under new identity  Most applicants are rejected   To join them, you need to be a young man, in good health, and buy a plane ticket to France. And be willing to kill people.  Here’s their web site  .  Foreign Service Exam  This is not as widely discussed and I should write more about it, but if you are interested in the US Foreign Service, the qualifications are:   U.S. citizens on the date they submit their registration package  At least 20 years old and no older than 59 years of age on the day you submit your registration  At least 21 years old and not yet 60 on the day you are appointed as a Foreign Service Officer   Note that neither foreign language skills nor a college degree are required. However, very few people pass their selection criteria. Prepare to do a lot of reading about being a foreign service officer  and accept that very few people pass the entrance exam. It takes a lot of study and discipline, but it’s one of many strategies available to you.  Fantasy versus Reality  When people as me what it’s like to be an expat, I often reply “how do you feel about immigrants”? It’s hard being an expat. Don’t underestimate homesickness. Many people want to be expats so they can tell their friends what wonderful and exciting lives they have but their friends are thousands of miles away. And the expat life is not glamorous. The French have a saying, « metro, boulot, dodo ». That means “subway, work, sleep”. It’s a typical “day in the life”. Being an expat means that you’ll get up, go to work, go home, have dinner, and sleep. That’s the majority of your existence at home and that will be the majority of your existence abroad. It’s the things outside of « metro, boulot, dodo » which make being an expat so rewarding, but those aren’t the bulk of your existence.  It’s not a magical, fairy-tale world out there, but it’s an incredible one. Immersing yourself in other cultures and understanding how other people see the world is one of the most wonderful things you can do in life.  Conclusion  If you’re young and determined, you have multiple avenues to live abroad, either permanently or temporarily. I would strongly recommend that you pursue several at the same time, maximizing your flexibility. Living in another country is not a right, it’s a privilege that you have to earn. If you’re determined enough, you can do it.  And keep reading this blog. As I research more ways to get out, I’ll post them here, along with my personal experiences. I’ve helped others leave their countries and I want to help you, too.  ","title":"Young Person's Guide to Moving Abroad","url":"\/blog\/young-persons-guide-to-moving-abroad.html"},{"body":"    The Continuum of Everything  What Is Life?  Life as Biopresentation  Objections to Biopresentation  Why is rethinking life important?  Is Biology the Continuum’s End?     The Continuum of Everything  I recently wrote about searching for extraterrestrial life in our solar system . In that post, I put forward a novel way of defining life that I call biopresentation . So it was with some surprise when I listened to a recent podcast of Isaac Arthur entitled “Post-Science Civilizations” and realized that part of what he was explaining reached right into the heart of my argument and even expanded it. Here’s the YouTube video for it:     In this, he discusses what would happen if we “run out of science,” though there are several possible paths to that. As someone who loves chess, one of the things I appreciated was him pointing out that knowing all the rules of chess doesn’t mean you’re a master of it. So if we learn all the rules of physics, that doesn’t mean we can instantly figure out what to do with them.  However, my big “a ha!” moment was Isaac Arthur pointing out that chemistry is emergent from physics and biology is emergent from chemistry. For now, we’ll avoid the obvious, and unanswerable, question: “what, if anything, is physics emergent from?”  Since chemistry emerges from physics, chemistry is physics. Since biology emerges from chemistry, biology is chemistry, but it’s physics, too. Now for some, this seems trivial. After all, we’ve learned from astronauts in microgravity that our body needs gravity to function well. But it turns out that life might not only require quantum mechanics to lead to chemistry, but life might be actively requiring quantum mechanics to continue existing. There’s an entire field called quantum biology  . We’ve known for over a decade that photosynthesis might rely on “quantum coherence” to generate energy  . Quantum neurophysics  is a new field, too. Special relativity may be critical to our functioning, not just to our existence.    Getting back to the main point, the question we should ask is, “if biology is emergent from chemistry, where do we draw the line?”  We don’t. It’s a continuum. Rather than refer you to a previous post I wrote, I’ll repeat myself.  What Is Life?  This must be prefaced with the “life as we know it” caveat (and I’ll give examples later). And I’ll come right out and say what’s on my mind: most definitions of life are a steaming pile of shit.  One definition of life  , as taken from Merriam-Webster, is “an organismic state characterized by capacity for metabolism, growth, reaction to stimuli, and reproduction.”   But right off the bat we see a problem. Mules, for example, don’t reproduce.   Would you say they’re not alive? Of course not. Biologists get much more sophisticated than Merriam-Webster, but many of their definitions still boil down to a true\/false value.  What about fire? It reacts to stimuli and grows  . We could even argue that it metabolizes—what it consumes for energy to keep growing—and reproduces by budding, but we don’t think of it as “alive.”  There are other properties of life, however. Homeostasis  is the body’s ability to internally regulate itself and keep functioning. Does fire exhibit homeostasis? It’s hard to argue that it does. However, warm-blooded and cold-blooded animals have completely different approaches to homeostasis and homeostasis evolved over time  , so there are different levels of homeostasis. Maybe we can argue that humans have a homeostasis level of ten while fire has a homeostasis level of one? A piece of flint, however, would have a homeostasis level of zero.  But why would we argue that?  Life as Biopresentation  When I was working on ETL  systems to reduce the cost of Phase III clinical trials  , I often found myself working with data supplied by many pharmaceutical companies. Company A might refer to “Dr. Robert Smith, St. Mary’s Hospital, Arlington, Virginia.” Company B might refer to “Dr. Bob Smith, St. May’s Hospital, Arlington, Virginia.” Are they the same person? “St. May’s Hospital” might be a typo. “Bob” is short for “Robert.” Maybe they are the same person.  In reality, it’s hard to definitively state “yes” or “no.” Instead, I had access to tons of information about each researcher, including email addresses, phone numbers, birth dates, and so on. I would assign scores (how reliable) and weights (how important) to each piece of data and if two researchers crossed a certain threshold, they were probably the same researcher and we didn’t have to spend valuable time and money trying to manually verify this.  This same approach should be taken to the definition of life. Instead of life being a binary “yes or no,” we should describe its “biopresentation,”   or “how present are the signs of life in a ‘thing.'” So for a virus:         Property  Score  Weight  Value      Metabolism  1  5  3    Homeostasis  0  4  0    Growth  0  2  0    Reproduction  3  4  12    Communication  1  5  5     In the above, a virus might have a biopresentation of 20. A human, however, would be radically different:         Property  Score  Weight  Value      Metabolism  5  5  25    Homeostasis  5  4  20    Growth  5  2  10    Reproduction  5  4  20    Communication  5  5  25     A human might have a biopresentation score of 100.    This would be particularly useful in describing abiogenesis. One hypothesis of abiogenesis is  :   Prebiotic atmosphere with ammonia, carbon dioxide and water vapor.  Lightning produces simple organic compounds that fall into solution in shallow water.  The compounds react further in a prebiotic broth, forming amino acids.  The amino acids link up with peptide bonds to form polypeptide chain proteins.  The proteins combine into more complex molecules that can replicate and metabolize simple substances.  Complex molecules and organic compounds form lipid membranes around themselves and start acting like living cells.   Traditionally, at step six is when we think “life”, but for biopresentation, we might starting thinking “lifelike” at step five. Maybe even four. Instead of life being a fixed point, it’s a gradual process. We don’t point to a thing and say “this is alive” because that gets us back to the arguments of what life is or isn’t. We simply describe its biopresentation.  This would be useful for biology today, too. Consider the endless arguments about whether or not viruses are live  . It’s often argued that viruses aren’t alive because they can’t metabolize outside of their host. That reduces metabolism to the primary (or sole) attribute of life.   But metabolism is the global term for anabolism and catabolism  . Catobolism is just breaking complex stuff into simpler stuff while anabolism is using that simpler stuff to do more complex stuff. Fire catabolizes, but doesn’t anabolize. Viruses do both. So maybe they’re alive? But some still reject that because, again, they can’t do that without a suitable host.  But what’s a host? Obligate parasites such as the protozoan Plasmodium   can’t survive outside of their hosts. Or consider humans. If I drop you into the middle of interstellar space without a space suit, you’ll stop metabolizing, too. So the context in which we can biopresent is also a spectrum. Viruses are starting to look more alive.  But wait! There’s more!  Most living beings have some ability to communicate with one another. Maybe all do. As it turns out, we’ve know for a few years that viruses can do this, too  . The human communication score might be at a “ten,” while viral communication might be a “one,” but fire’s communication is “zero.” The weight of this factor is probably pretty high, so fire’s biopresentation would be pretty low, but viruses could be easily crossing the threshold.  Pragmatist that I am, I want to skip the philosophical debates about whether or not something is alive and go with the pragmatic discussion of how closely something resembles life. However, we can’t ignore the philosophical implications.  Objections to Biopresentation  At this point, I can see many biologists howling at the above. In a 2013 Wired article entitled It’s (Almost) Alive! Scientists Create a Near-Living Crystal  , the lede reads:   Three billion years after inanimate chemistry first became animate life, a newly synthesized laboratory compound is behaving in uncannily lifelike ways. The particles aren’t truly alive — but they’re not far off, either.   The crystals created had metabolism and mobility, but couldn’t reproduce. You can read the original study at The Royal Society website  .  For these crystals, one could argue that a), they’re not alive, or b), what their relative biopresentation level is. Some people would be uncomfortable describing crystals as alive, but if we’re to search for life elsewhere in the universe, are we going to argue that life which doesn’t share the same biochemical basis as our own isn’t alive?  Or consider self-replicating Von Neumann machines  . By the biopresentation argument, they would likely score very high, but are machines “alive”? Or apply biopresentation to The Gaia Hypothesis  . Is our planet alive? That’s definitely a philosophical question, but in light of biopresentation, depending on what factors to include, we could make a strong argument that it is.  Why is rethinking life important?  If we go with a binary “yes\/no” for life, it means that we have defined a limited, possibly incorrect, set of properties necessary to define life. Something not meeting those conditions might be disregarded. If, however, biopresentation is adopted, we can start defining “lifelike” properties and perhaps not be so quick to dismiss something that is otherwise alive.    Is Biology the Continuum’s End?  If biology stems from chemistry stems from physics, is that the entire continuum? I don’t think it is. I think it’s pretty hard to describe viruses as being very sentient (the ability to feel and perceive things), but fish are. By the same token, it might be hard to describe fish as being particularly sapient (the ability to learn, self-reflect, and so on), though again, both sentience and sapience are probably continuums, as are most things complex.  So clearly biology can lead to sentience and sapience, but do they require biology? Possibly not. Consider the recent case of Blake Lemoine who contended that some of Google’s software had become self-aware  . While I tend to agree with Google that Lemoine was wrong, I am not so sure how long I can hold that position in the face of greater AI advancements. Since this is just electrons flying around silicon, can we argue that sapience doesn’t require biology? If so, the continuum could be across a tree or graph instead of a line. Or since humans built this, does that make the argument moot? I don’t think this is possible to answer, but it raises other interesting questions.  If we have a complicated chemical reaction that biopresents as zero (“pure chemistry”), could sentience or sapience emerge from that? Could sapience emerge from pure physics? This gets into the metaphysical realm and we cannot answer that. It starts teasing out hints of a god or gods. While I am an atheist, many of my friends are not and these question approach the realm of the ineffable, a term frequently applied to God. I worry that I sound like the atheist equivalent of Reverend Teilhard de Chardin  , a Catholic priest who tried to unify science and religion.  ","title":"Redefining Life as a Continuum","url":"\/blog\/redefining-life-as-a-continuum.html"},{"body":"    Introduction  The Drake Equation  (Not) Detecting Life  Assembly Theory  What If We Find Life?     Introduction  Few questions intrigue us as much as “are we alone in the universe.” Like many, I’m obsessed with the topic. So obsessed, in fact, that the National Library of Finland released an hour-long talk I gave on the subject.     I’ve also previously written on this topic and the above talk is an in-depth expansion of what I’ve written earlier, but hardly scratches the surface. But to understand a bit more about whether or not there is life elsewhere in the universe, let’s look at some of the background.  The Drake Equation  The Drake Equation is a famous probablistic equation created by Dr. Frank Drake in 1961. In this equation $N$ is the estimated number of intelligent civilizations we might be able to detect.  $$N = R_* \\times f_p \\times n_e \\times f_l \\times f_i \\times f_c \\times L$$   $R_*$ – Average rate of annual star formation in the Milky Way galaxy  $f_p$ – Fraction of those with planets  $n_e$ – Fraction of those planets which can support life  $f_l$ – Fraction of those which develop life  $f_i$ – Fraction of those which develop intelligent life  $f_c$ – Fraction of those generating detectable signals  $L$ – Lifespan of those signals   The variables $R_{*}$ through $f_{c}$, when multiplied together, tell us how many intelligent civilizations “mature” per year, to the point where they can be detected. Multiply that by $L$, the last term, and you have an estimate of the total number of intelligent civilizations we might detect at any one time.  At the time the equation was created, we didn’t know any of those numbers, but Drake himself estimated that $N$ was around 10,000  . Today, we have been slowly knocking those numbers down, one by one. Of those we think we know the what the values are, we have:   $R_*$ – Average rate of annual star formation in the Milky Way galaxy (6-7)  $f_p$ – Fraction of those with planets (50% to 100%; probably closer to 100%)  $n_e$ – Fraction of those planets which can support life (1?)   The last number, “Fraction of planets which can support life”, is often reported as $1$, but we already know that many moons in our solar system might support life . So we could easily have multiple “planets” per system which could support life. We think there are between 100 to 400 billion stars in our galaxy  , so that could easily lead to over a trillion heavenly bodies which could support life in our galaxy alone!  We don’t know how many stars are in our universe, but current estimate suggest there might be between $10^{22}$ to $10^{24}$ stars  . That’s up to 1,000,000,000,000,000,000,000,000 stars!  So what about the number of planets which develop life, have intelligent life, and so on? We have no idea what those numbers are. However, that doesn’t mean we know nothing about those numbers. We exist, Therefore, all of those numbers are greater than zero. The first, $f_l$ (fraction of planets which develop life) is what we’ll focus on here because if that number is too low, none of the others matter.  (Not) Detecting Life  You may have read in the news lately about a new method of detecting life on Mars: DNA sequencing at the picogram level to investigate life on Mars and Earth  . This is remarkable work, but to point out the obvious—which the authors of the work are certainly aware of—what if the alien life doesn’t have DNA?   In a paper entitled Is There a Common Chemical Model for Life in the Universe?  , by Benner, Steven &amp; Ricardo, Alonso and Carrigan, Matthew. (2005), point out that most “universal” probes for life target the ribosome, present in virtually all life on Earth. Again, what if the life doesn’t have ribosomes? Using this as a tool to detect life won’t work and we know nothing about alien life (if it exists), so can’t make strong statements about what that life is likely to be.  There might be another approach, though. Two Viking landers arrived on Mars in the 1970s and their labeled release experiment might have detected life . In short, in separate experiments conduction over 6,000 kilometers apart, the landers injected Martian regolith (the surface material on Mars) with a radioactive nutrient solution and something interacted with the solution and released the radioactive carbon-14. It looked an awful lot like metabolism. It might just have been a chemical reaction. We don’t know.  What’s interesting about the Viking Lander experiments is that they weren’t searching for specific chemicals. They were looking for a biosignature indicative of life. Yes, we made assumptions about the “nutrient” solution, using seven molecules found in the Miller-Urey experiment  , but trying to have an agnostic approach to detecting life was promising. That brings us to Assembly Theory.  Assembly Theory  Detecting alien life is a tricky business because:   We don’t know if it exists  If it does exist, we have no idea what it’s composed of   Assembly Theory was first proposed by Professor Leroy Cronin  , the Regius Chair of Chemistry in the School of Chemistry at the University of Glasgow. Instead of just asking what a molecule is , he asked how it came to be. In other words, how was it assembled? He’s established the notion of a “molecular assembly index” (MA) which represents the minimum number of steps necessary to assemble a given molecule.    Consider the word “going.” Imagine that each letter is an atom floating around which can freely attach to other atoms to form words. The MA of a single letter is zero because it already exists. However, when you add the letter “o” to the letter “g”—forming “go”—you have an MA of 1, because one step has occurred. When you then add the letter “i”—forming “goi”—you have an MA of 2, and so on. The word “going” therefore has an MA of 4. Of course, by the constraints we’ve listed, all other possible combinations, such as “oin” (MA of 2), “in” (MA of 1), and so on, are floating around in our primordial soup.  But consider the word “hodgepodge.” By the time we’ve assembled “hodgep”, we have an MA of 5. You might think the word “hopgepodge” has an MA of 9, but remember that substrings of “hodgepodge” are floating around, too. One of those substrings is “odge” and can attach to “hodgep” in a single step, giving us a final MA of 6, not 9.  Of course, chemistry is much harder than spelling. Molecules are 3D structures and individual atoms don’t form bonds randomly with all other atoms. But you get the basic idea.  In the paper, Identifying molecules as biosignatures with assembly theory and mass spectrometry  , (Marshall, S.M., Mathis, C., Carrick, E. et al.), Professor Cronin and his colleagues took the time to:   calculate the complexity of several million molecules and validate that their complexity can be experimentally determined by mass spectrometry.   Their findings?   These results demonstrate that we can identify the living systems by looking for mixtures with an MA greater than a certain threshold. In the case of the analysis presented here it appears that only living samples produced MA measurement above ~15.   In short, molecules with an MA above 15 were invariably produced by life, while molecules below that might be produced by both abiotic or biotic processes. If these results hold, we might have a “universal” method of detecting life, with the caveat that if there’s life that only generates molecules with lower MA numbers, we’ll still not detect it. However, this approach seems rather promising.  To give some context, sometimes you’ll read breathless news articles proclaiming that we’ve found “complex” organic molecules in space. For context, organic molecules are just molecules with carbon-hydrogen or carbon-carbon bonds They’re all over the place because carbon is one of the easiest molecules to form bonds with. One of those “complex” organic molecules found in space is ethyl formate. It has an MA of 3. The molecule n-propyl cyanide has also been seen in space. It also has an MA of 3.  In fact, most complex organic molecules in space have rather low MA numbers and it’s not hard to imagine abiotic processes producing them. On the other hand, we may have detected tryptophan around the IC348 star system  , about 1,000 light years away. Tryptophan has an MA of 11.  (If you’d like to check these numbers, I wrote a script to do that for you  , but it requires the Molecular Assembly REST API  . That doesn’t always seem to return results. In correspondence with Dr. Cronin, he confirms they’re working on a new version)  Throughout all of the above, keep in mind that Assembly Theory, like all new scientific endeavors, has its detractors.  What If We Find Life?  Three things. Given that there are possibly a trillion or more candidate places for life to come into being in just our galaxy, given that we know $f_l$ is greater than zero, and given that we’re seeing more and more complex chemistry in our own solar system which would be easy to explain biologically (were we to know the biology is there), I’m gradually leaning to the idea that we are probably not alone in the universe. Of course, that’s a hunch, not science.  In the fifth century BCE, the Greek philosopher-scientist Anaxagorus speculated that there was life on the moon  . Many have speculated about this throughout history and certainly in pre-history. That we have so many religions asserting non-human intelligences with whom we can communicate is part of this longing of ours to know that we’re not alone in the universe.  Finding life, but not intelligent life, will not be enough to satisfy us, but it will be an amazing step forwards in filling in the last of the variables in the Drake Equation. Further, if we find life in our own solar system, but it has a different origin from our own, it’s virtually impossible that the only two abiogenesis events in the universe happened in the same solar system. The universe would be teeming with life.  For those wondering about abiogenesis (the origin of life from non-life), here’s a video by Sabine Hossenfelder describing our current thinking on the topic.     ","title":"Searching for Alien Life","url":"\/blog\/searching-for-alien-life.html"},{"body":"  I wrote this a while ago, exploring first-person, present-tense, non-linear story telling. That wasn’t the intent, but when the story came to me, it was there and it wouldn’t go away. I don’t say it’s a great piece, but I’m proud of it. I also find that first-person, present-tense makes it easier for me to avoid filter words  . I have no idea why. I have a chapter from my novel that’s written in first-person, present tense, and it still gives me chills to read it. Narcissism is fun!  It’s worth noting that the main character is based on a friend. We worked together and he told me what it was like growing up black, in a gang, in Portland, Oregon. I’ve taken a few liberties, but his voice rings strongly in my mind and I think I’ve been faithful to it, though I’ve toned it down. A white author writing about black experience cannot use the n-word.  The \" now \" and \" then \" are recent additions by me because some early readers complained it was too difficult to follow the non-linear story line. However, I can’t simply re-arrange the story in a linear fashion because that destroys it. Hence, my added cues for the reader. If they don’t help or distract, let me know in the comments. In fact, I’d be grateful for any feedback you care to leave. I want to be a better writer.  Also, I know it’s the US Marshals who handle the witness protection program. I wrote “FBI” because I wasn’t sure most people would know who the US Marshals are.  The Names We Never Say  Now.  “Parents died in a house fire when I was a kid, so no, I don’t have no childhood photos.”  That shuts ‘em up. It always does. I roll up my left sleeve. A mess of white scar tissue on black skin.  “That’s my childhood photo.”  I got that from a night drinking and messing with a barbecue, but I didn’t tell them that. Heck, I shouldn’t even have shown them the scar, but lying always made me nervous and I felt like I had to back it up. Old, stupid habits die hard.  “James, I’m so sorry.” Frank’s squeeze—don’t even remember her name and there’s no point because she’ll be gone next week—goes beet red with embarrassment. I feel sorry for her. Frank’s my best friend now, but he’s a bit of a bastard about women. It was the only thing we argued about. I’ve found God now. I think. Not sure if he’s found me.  “No worries, girl. You didn’t know. Gonna get another beer and if you two ain’t drinkin' with me, I’m gonna be mighty offended.”  Middle of a Saturday afternoon and Frank’s here to watch the game with me. My wife and daughter are out on a mother\/daughter field trip and I have a day off to watch LeBron destroy Miami. It’s a hot summer day and a few Coronas with lime go down a treat. I’m in the kitchen, reaching in the refrigerator when I hear the clang of the mail slot in the front door.  I walk back into the living room and there’s Frank, reading an envelope.  “Who’s Jamal?”  With that innocent question, my life is over for the third time.   Then.  You have no idea what it’s like growing up near Martin Luther King Avenue in Portland, Oregon. I had the luxury of traveling a lot before I got busted the second time and every place that has an “MLK” Boulevard\/Avenue\/Promenade—nothing ever as pedestrian as “Street”—is always through a poor black neighborhood and that’s where I grew up. That’s where I dealt. That’s where I joined a gang. That’s where I lost my virginity and, as a result, acquired my first scar. A stab in the gut courtesy of her boyfriend.  I’m not stupid. I didn’t brag to nobody, but one night hanging out, there he was. His name was Aaron and he walked up, smiled, and said “‘Sup, Jamal?”  I just nodded and he slid a knife in my gut like it was the most natural thing in the world. To him, I suppose it was. People were screaming and Aaron just walked away.  I told the cops I didn’t know who it was. You settle your own shit. Course they sent a black cop round. Like I’m gonna trust one of them. I went looking for Aaron later. I thought I sort of deserved what I got, but you gotta save face, right? Everybody said Aaron disappeared. No idea where he went off to.   Then.  I didn’t plan to grow up dealing; I wanted to go to college and study literature. Momma didn’t have much, but she loved books. Naturally, I loved books too, but lettin' on that I huddled under the blanket, switching from Heinlein to Faulkner to “Poets of the Nuyorican Café” made me a joke. So publicly, I read biographies of Malcom X, Nelson Mandela, or anyone else that wasn’t going to piss off my friends too much, though I liked them, too.  But nobody hired boys named “Jamal.” At least not good jobs. At least not in the whitest major city in the good ol' US of A, love her or leave her. Which is how I got busted the first time, dealing while working at a car wash. I wasn’t stupid and I’ve got a feel for people, but it’s not perfect. The undercovers dropped a name and asked if they could score. Later they lied their lilly-white asses off in court. There was no entrapment; I offered it to them. Guilty.  But I never grassed on no one. You don’t get two years for a first offense, but with a name like “Jamal” and refusing to cooperate, I had a DA in a bad mood. Got out six months early for good behavior, hit the streets and life was over.  I tried to go straight. I really did. But when you have to explain time in prison, you don’t get no real jobs, which is why I felt real lucky to have Christine. She was Aaron’s ex, the one I slept with. He left her a broken nose as a parting gift, but she was lucky and it healed sort of upturned. It was cute, but I hated Aaron. You don’t hit women. No way.  Christine was a receptionist at a car dealership and got me a job as a salesman. You piss clean in a cup and talk like nobody’s business and they don’t care about your past. They just want to know if you can make them money.  Christine was special. For the first time in my life, I felt like things were going good. I didn’t feel too comfortable round her mom, though. She didn’t care about my past—too many good boys get caught up in bad things—but she creeped me out. I was sitting there, sippin' iced tea, waiting for Christine and talking to Christine’s mom, Gisele. “Gee-seh-luh.” What sort of name is that?  You didn’t lie to Gisele. I learned that early on. I never felt good about lyin'. I could lie when I had to, but not with Gisele. She was a human fuckin' lie detector. I once told her I sold 18 cars last month. I just wanted her to know I could take care of Christine. That’s exactly what I was thinking. Gisele just looked at me and said “I know you’ll take care of Christine, but you didn’t sell no 18 cars last month, Jamal.”  She could have found out how many cars I sold, but how the hell did she know what I was thinking? I never lied to her again.  So there we were, drinking iced tea, and I said Clinton was going to crush Trump in the election. Gisele looked sad and said, “no, she’s not.”  And just like that, I knew Clinton would lose. Gisele did that to you.  When Christine got home that night from college, she stepped into the living room and plunked herself down on the old, ratty sofa and curled up next to me. I wrapped my arms around Christine and knew with dead certainty that I loved her. Probably the only reason Gisele let me in the house. She knew.  And then she didn’t let me in no more. I wasn’t a great salesman, but the other guys knew my past and kept asking if I could hook ‘em up. That’s how I got back in. I wanted to make money for Christine and here it was. Just one look at her big, brown eyes and I’d have given her the world if I could. So I lied and said I was making good commission on my cars, when really I was hooking up the other salesmen. And then I started hooking up more than other salesman. That’s how it is.   Now.  “Jamal? He’s my cousin in San Francisco. Don’t know why I’d be getting mail from him.”  Frank hands me the envelope and I look at the address. “Jamal, c\/o James Hillman.” I’m still Jamal inside, but Frank and everybody else knows I’m James. Except for my wife, Angela, née Christine. Even our daughter, Sarah, don’t know our real names. I hated lying to her about it, but I love her as much as Christine. No way in hell I’m going to give her a gun to take her parents away from her.  I put the envelope aside and Frank forgets about it. Every nerve in my body is screaming, but I wander back to the TV, acting like nothing was wrong. I still don’t know who won that game.   Then.  New apartment. New king size bed. Mind wandering.  “What sort of name is Gisele? I never heard no name like that.”  Christine was lying in my arms, our bodies still covered with sweat, and whispered to me, “her name’s not Gisele.”  “What the hell does that mean?”  “She has this thing about names and won’t tell anyone her real one. Says it gives people power over you. And stop talking that way, you’re better than that.”  “So what’s her name?”  Christine sat up and looked down on me.  “No, Jamal,” she said. “She maybe has weird ideas, but she’s my mother. I won’t dishonor her.”  Christine was sort of a Christian and honoring her mother and father (God rest his soul) was important to her. I say “sort of a Christian” because she knew I was making more money than selling cars could provide, but she never asked. I liked to think it’s because she loved me, but I never asked, either.  I sat up and wrapped my arms around Christine. I knew not to press about Gisele’s name and hugged Christine tightly, but this bothered me. Everything about Gisele gave me the willies and at this point in my life, anything I didn’t understand was a threat. Except I didn’t understand Gisele and she didn’t feel like no threat.   Then.  A couple of years before that, I got some really bad good news. I was a small-time dealer, now selling cars and coke and my supplier invited me to dinner on a night Christine was in school. I went round to the restaurant, but my supplier wasn’t there. Lamonte was. Lamonte was my supplier when I went to prison. I never dropped his name, but maybe he didn’t know that. But when he saw me, he smiled, and waved me over. “Sit down, Jamal. It’s time to talk about your future.”  I looked round, but no one in the restaurant said “cop” to me. Not even Lamonte. A waiter walked up and smiled, so I sat. I ordered a beer, took a menu, and waited for the bad news.  “I hear you play straight. And I know you do. You could have sent me up, but you went to jail rather than drop my name.”  “Prison. Jail may be shorter, but at least they got shit to do in prison.”  Lamonte nodded; he knew what I meant. You don’t, but he does. In jail you sit on your ass and play cards, waiting a few months to get out. You’re bored stiff. In prison, you’re there for a long time, so they have weight rooms, college classes, libraries, you name it. Jail doesn’t have shit, but in prison, they give you a chance to better yourself. Maybe you’ll come out with a degree. Or maybe it’s a crime finishing school, with professors who were stupid enough to get caught.  The waiter came by and I ordered. When he left, Lamonte leaned forward and said “You had my back when you didn’t have to. Now I got yours. You say the word and you don’t have to sell no more god-damned cars. I’m doing good and I want people I can trust. I trust you.”  I sat back, scared as hell. Christine looked the other way about the extra cash I was earning, but that’s only because I had a job. She was the one good thing in my life and I didn’t want to fuck it up. That’s what I told Lamonte.  “You’re a good man, Jamal. I appreciate what a good woman can do for a man, but what can you do for her? You’re a felon. You know you’re never going to get a real job. Oh, you might be a sales manager some day, earn whatever small bit of green the man wants to toss your way, but you really want to take care of her? I can offer you that. It’s a real job. I start you at five grand a month and you run some dealers. I won’t lie to you. I ain’t promisin' nothin' bad’ll ever happen, but if it does, we got each other’s backs. And you treat us right, we treat you right. You can keep Christine in college and have nice things.”  And that’s how life ended the second time, but at least it paid well.   Now.  It’s two days before I open that envelope. I know it’s stupid, but I’m scared. I don’t know what the hell to do. Christine—excuse me—Angela probably knows something is up but I tell her it’s just work. With the economy the way it is, she accepts that. I hate to lie to her, but I’m really thinking about Sarah. When she was born, she brought joy to our lives like we had never experienced. Oh, we were as happy as our circumstances could allow, but this felt like it was for real this time. Past sins were absolved and beautiful Sarah laughed and cried and hiccoughed and I even started going to church. My life was almost complete. Even changing diapers had made me smile.  Sarah had never known any life but what we had. I didn’t want to take her out of school, but there’s an envelope in my hand. Hell, I like going to PTA meetings. I want Sarah to have everything I didn’t. And so did Angela. We have the life we always wanted. It isn’t perfect, but it’s stable. And always with that tinge of fear. Now I know I should call the FBI and I know they will take away our life and give us a new one. Sarah will cry for her friends in school and I have no idea what to say to her.  I don’t know what to expect when I open that envelope, but I certainly don’t expect what I find.  I unfold the paper and read. Or at least I thought I would. The paper is blank. I turn it over, but nothing is there. Paranoid, I look in the envelope for weird powder or whatever the hell anthrax looks like, but I see nothing. I sniff the air. Nothing. I sit. I wait.  I have tears running down my face when I notice the writing on the back of the envelope.  “Gisele.”  And then the mail slot clanged. There was another envelope on the floor.   Then.  Life working for Lamonte was good. Lamonte was careful and if a dealer started using, they were out. You couldn’t trust them. I was one of several “account managers” and we were a tight group. We knew Lamonte’s star was rising and we along with it. Because I talked proper when I needed to and didn’t mind wearing a suit, I soon found myself working directly with several favored customers. You’d probably even know the names of a couple of them, but you won’t hear them from me. When I landed a narc as a customer, we had a nice celebration. Portland cops had worked real hard to clean up, so getting inside news was a real score. He got his shit for free, sometimes wrapped in Benjamins, and we heard rumors nobody else heard.  That’s how, when Lamonte got busted, I had his job. Lamonte was true to his word and didn’t name names. And Christine didn’t ask questions about my import\/export business and cried when she thought I wasn’t looking. I was a “district manager” and the account managers reported directly to me. I didn’t change nothin'. What worked for Lamonte worked for me.   Now.  I open the envelope right away this time. A single sheet of paper. A single word. “Aaron.” Written on the back of the envelope was “Gisele,” in Gisele’s handwriting.  Of course it wasn’t Gisele. She died years ago.  I’m staring at the envelope and I dimly remember that like Angela and myself, she had a different name. How many people knew that? Sure, she had different reasons, but still. She was buried years ago and now I have this envelope in my hand and a sheet of paper which reads “Aaron.”   Then.  I met Aaron again shortly before my second bust. I was pulling down a cool seven grand a month. I never skimmed, I didn’t cut the dope too much and when I fired a dealer for using, I gave them severance pay out of the expense account. Then I told ‘em to get lost.  One day on an exchange, Big Boss told me to carry. We were dealing with some new boys. They were unknown, but they had lots to move. Normally I don’t hold no truck with guns, but when the big boss trusts you to watch his back and you’re looking to move up, you do what you’re told. So I carried. And there, in an old warehouse, was Aaron providing muscle for the new boys.  I slowed down. I guess the big boss sensed something was up and he slowed down, too. Then they slowed down. All of a sudden, a clean exchange became real tense. I could see Aaron’s eyes moving from man to man, doing the math. I wanted to hurt Aaron bad. Not for what he did to me—I could forgive that—but for what he did to Christine. But I can do math too and the numbers didn’t add up. Aaron and I just stared at each other, hatred in our eyes, but the deal went down.  Big Boss later asked me what happened but I told him I just had a bad vibe. I stuck to the story. No sense worrying the man. Aaron must have lied to his big boss too because we had more deals and I always carried. You don’t need to tell me twice.  When Aaron finally came looking for me, he shot the big boss and I killed Aaron. Or at least I tried to. Lucky shot got him in the leg. The Feds offered a deal: it was self defense if I testified, Christine and I would get the Witness Protection Program, provided I kept myself clean. If I didn’t testify, I’d go down.  I don’t grass, but I had Christine to think about. It was a hard decision. I tried to keep my account managers out of it, but no deal. And I had to point out the big bosses. Mine survived being shot and Aaron’s boss went down, too. One stupid fool and everybody’s looking at me. Well, everybody except Aaron. The slippery son of a bitch was offered the deal, too. He took down his side, I took down mine.  So Christine and I live in hiding. New ID cards. New social security numbers. I’m a district manager of a snack vending company—yeah, I get the irony—and Christine, bless her, didn’t leave my side. She was just happy I was out.  And then we had Sarah.   Now.  You’d be surprised how many people die in the Witness Protection Program. The FBI drilled this into us repeatedly. All it takes is one moment of homesickness, one quick phone call, and if the people who want you want you bad enough, you’re dead. One fool went back to his old home and turned the doorknob and it blew up.  Christine and I knew the score. Neither of us had anything to go back to. Christine became Angela and I became James. We were moved to a new city in another state. The FBI got me a job, a small house, and a phone number for emergency use only. No friends. No social circle. No way back. Pretty fuckin' shitty deal, but it beats dying.  But we had each other. That’s enough.   Then.  Gisele died unexpectedly. She was in great health and whenever Christine wasn’t with me or at school, she was with her mom. So when one day Gisele didn’t answer her door, Christine panicked. She just knew something was wrong. Shortly after, we were at Gisele’s funeral. Brain hemorrhage. The doctor promised us it was painless and looking at her in the casket, she looked peaceful, but then, I guess that’s the mortician’s job.   Now.  I’m looking at the sheet of paper and know I have to go to the FBI now, but then there’s Sarah. What do I do about her? She’s not old enough to know what’s going on and there’s no way she’ll understand if we call her a different name. She won’t know why she has to leave her friends at school and Angela’s going to be terrified. So I do what I have to do. I buy a gun.  Normally, ex-cons don’t get to buy guns but James Hillman is not an ex-con. I buy a shotgun to avoid the waiting period and I go home and head down to the basement. I turn on the light bulb and ignore the damp, musty smell. I open my toolbox and a few strokes of a hacksaw removes most of the barrel. A bit of work with some files and cutting down the stock and I have a nasty short-range weapon. A nasty short-range weapon that’s pretty easy to hide. Pretty illegal, too, even if James Hillman isn’t an ex-con. I’m going to have some trouble explaining this when I get caught.  I will get caught. I know that, but I don’t care. But Christine and Sarah will be safe. I really have no choice.  I think about this a lot and I figure that if Aaron’s after me, he’s a lone nut. Otherwise, Gisele would have said something different, right? But no, Gisele’s dead. Who the hell sent that damned envelope?  Aaron’s Lee Harvey Oswald and I’m John F-in' Kennedy. Except JFK wasn’t married to Oswald’s ex. JFK didn’t know Oswald was coming to seek revenge for JFK being with Jackie. And I will kill Aaron and the cops will take me away and Christine and Sarah will be safe. That’s all I care about.   Now.  I stop staying late in the office and bring my paperwork home with me. Christine—or is she Angela?—asks what’s going on and I tell her I just want to spend more time with her, even if I have to work. I thought I caught her crying once. She knows something is wrong just as she always does.  Sarah likes to crawl into my lap. I kiss her forehead and keep working, my mind trying not to think of the sawed-off shotgun in my briefcase. On a hot Saturday, I’m not surprised when the third envelope falls through the slot. They always arrive when Angela\/Christine and Sarah\/Sarah are out.  I open it and read two words, “Tomorrow Night.” On the back of the envelope was a single name, “Lizbeth.”  That night, curled up next to Christine\/Angela, I ask her a simple question.  “Her name was Lizbeth”, Christine murmurs.  But by now, I already knew that.   Now.  Sunday. Sarah’s best friend was always asking her to sleep over and, for the first time, I said it was a good idea. I am over-protective of Sarah. Christine thought Sarah staying with a friend was wonderful and had already started to plan a romantic evening but an emergency phone call from work took me away.  And an emergency phone call from Frank took Angela away, too. He says he’s broken up with his girlfriend and is upset, but since I’m not there, will Christine come? I told him to keep her away the entire night, whatever it took. I’ll be happy if she’s mad. Dead people don’t get mad. As she drives off, I huddle down further in the bushes, the shotgun in one hand and a cell phone in my pocket.  Aaron shows up just before midnight. The lights are off and as he limps to the front of the house, I watch him, quiet as the names we never repeat any more. He walks up to the front door and feels under the mat for a key. That’s good because I left one for him. If he breaks in, maybe I can get away with shooting him in the back. As he carefully turns the key in the knob, I raise the shotgun.  And then I put it down again. Good people might do bad things, but I couldn’t do this. Aaron tiptoes inside and I lift the cell phone and call the special number the FBI gave me for situations just like this. And that’s when Christine pulls back into the driveway. And that’s when Aaron steps outside. And that’s when Aaron pulls a gun and points it at Angela.  Or is she Christine? I’m confused. Gisele’s in my mind telling me I’m a good man who does bad things. I’m not. I’m a bad man trying to do good things.  I shoot Aaron in the back, rack, shoot him again, rack, shoot him again and keep going until the shotgun is empty. Christine is screaming.   Now.  You might think I’d get in trouble with that shotgun, but when you’re in the Witness Protection Program, some problems go away. “Brave Homeowner Kills Intruder Threatening Wife” read the headline. Any publicity usually means you need to leave. Publicity and knowing someone has found you guarantees it. I don’t know what we’ll tell Sarah, but the FBI tells me they already have a plan for that.  When we move into our new house, there’s an envelope lying on the floor in the foyer. There’s no name on it, but it has a single piece of paper with two words, “Thank you.”  ","title":"Short Story: The Names We Never Say","url":"\/blog\/short-story-names.html"},{"body":"  Introduction  I’m an American who’s lived in the US, the UK, Japan, the Netherlands, and now France. People kept emailing me over the years to ask me how to move to a foreign country. As it turns out, the answer is complex and I got tired of cutting-n-pasting the same email over and over again. So I started a blog entitled “Overseas Exile.” I posted several times a week for many years, but I let the domain lapse and now someone else has snapped it up. You can still read my old posts  , but I felt, given the current climate in the US, that it’s worth sharing some of this here.  This wasn’t my first post, but it was one of the most important, despite the silly premise.  Count von Europe  So your eccentric great-aunt Gertrude has invited you over for dinner and when you arrive, you find a mysterious stranger with her. She introduces him as “Count von Europe”. After a long and pleasant conversation over food and drinks, the Count says “your aunt invited you over because I need someone to watch my Bavarian castle while I’m away for a year. I’ll also pay you €50,000 a year and let you borrow the Bentley when you want to travel around Europe on your days off. When can you fly to Germany?”  Sadly, for many people I’ve spoken with, their “dream” of living abroad is little more than “Count von Europe.” I see people on message boards saying things like “I want to move to Italy. Anybody got a job\/marriage\/house for me?” Honestly, it’s not going to be that easy. However, you can do it if you plan things right and understand what’s involved. When I talk to people about this, I sometimes use “Count von Europe” as a thought experiment, but the ending is a little different. He makes the same offer, but with a catch: you have to leave tomorrow.  First, we’ll have a slight digression. Many years ago I used to sell cars (don’t hate me for that. There are plenty of other things you can hate me for). One thing management drilled into our heads over and over again was that 80% of people only agree to drive home in a car after being asked to buy five times.That’s because people have objections. “The price is too high.” “I wanted red, not blue.” “I’m just looking.” My job as a salesperson was to understand and overcome all objections the customer had. If I could do that, they’d agree to buy the car, but it’s harder than it sounds.  Objections  If you want to take up Count von Europe’s offer, overcome your own objections. If you can do that, you’re one step closer to moving to another country, so let’s look at those objections.  I don’t have a passport.  This one boggles my mind because I’ve been surprised at how many people say they want to live in another country but don’t have a passport. If you’re from the US, go here to apply for a passport  . Otherwise, it’s fairly easy to find out where to apply.  I’m locked into a long-term lease.  Then find a way to get out of that lease. When I was offered a job in another country in 2001 I had three months left on my lease. I went to the landlord and explained the problem. She was actually very gracious about it and let me end the lease early without penalty. It never hurts to ask!  Failing that, convert your lease to a monthly lease when it ends. Or have enough cash on hand to buy out the lease. Or when it ends, become someone’s roommate.  But I own the property I live in!  Then sell it. Or rent it out and rent a room somewhere. Or find a property management company who will rent it out for you.  I have a wife\/husband\/partner\/children, etc.  That’s a tough one. Right now you have to decide what you want to do. If you have obligations to others who don’t want to leave (I’ve been there and it’s tough), then you’re stuck. I can’t\/won’t offer advice here other than to suggest taking them on a vacation to the target country. It’s a lot easier to appreciate something if you know something about it.  I have a cat\/dog\/iguana.  You can still move overseas, but you have to make sure you conform to your target country’s regulations for shipping your pet there. The “Transitions Abroad” Web site has a good article to help you understand the basics of moving a pet overseas  .  Other Objections  If you get a chance to leave tomorrow, what’s stopping you? Figure out what that is and if it’s more important than living abroad. And fix it.  Conclusion  Think about these objections and any others you may have. This is the starting point for being able to move overseas. If you find that you could say “yes” to Count von Europe, you’ve gotten enough of your life in order that you can make this happen. This is key to realizing the dream of moving abroad. I don’t expect that you’ll actually be able to leave at the drop of a hat, but if you can, it’s much easier. Think how much cheaper and faster it is to move to a foreign country with a backpack as compared to an entire household of goods.  ","title":"Why You Will Say \"No\" to Living Abroad","url":"\/blog\/why-you-will-say-no-to-living-abroad.html"},{"body":"    My First Weddings  Why I Stopped Officiating Weddings  Why I Started Officiating Weddings Again  The BBC     My First Weddings  Note: Much of what follows is from memory, but the ‘seventeen weddings' described near the end are from local copies of old blog posts I had written on the topic, many years ago. Hence, the detail.  I have, to date, been an official participant in 21 weddings in the US since I turned 18.   Twice I was the groom. The other 19 times, I was the minister. The story gets weirder from there.  My first wedding was three decades ago. It lasted five years and she and I grew apart. The divorce was difficult, but then, they often are. I hope she’s doing well.  My twenty-first, and last, wedding was in Tower Bridge, London, on June 20th, 2010, to my lovely wife Leïla. We’re still happily married, with a beautiful daughter, Lilly-Rose.    Our wedding. Source    Being a husband and father has been exhilarating, wonderful, and sometimes exhausting, but I wouldn’t trade my wife and daughter for anything.  Getting back to the wedding, the date, June 20th, was selected because it’s my birthday. I have no excuses if I forget. She knows me too well and I can’t fault her for that.  Curiously, this is my second wedding on my birthday. The first “birthday” wedding was also the first wedding I officiated at. My friends, we’ll call them Alice and Bob, called me, frantic because they lost the judge who was going to perform their wedding at a lake. Apparently he was retired, senile, and had no recollection that he was to perform their wedding.  I, however, was neither old nor senile (I’ll not swear to either today), but I am an ordained minister. Despite being an atheist, I was ordained by The Universal Church  in September of 1990. Back then, you had to do it the hard way by sending them a postcard with your details. Today, you can do this online.  Because I was an ordained minister, Alice and Bob explained the problem and asked if I could officiate at their wedding. I told them I would call them back after I found out. I called my church and they agreed to Fedex™ the paperwork to me, free of charge. I called my friends and told them I’d be honored to preside at their wedding. I neglected to tell them it was my birthday; that would have been bad form.  I filed my paperwork with Multnomah county, in the US state of Oregon and was quickly approved, though not without some odd looks. I was wearing a pair of black earrings and a silver\/grey shirt that buttoned up the side, just the sort of thing I’d wear for a Goth Night of clubbing. But Portland being Portland, odd looks were all that came of that.    The first wedding I officiated at.   The wedding at the lake went off without a hitch and I got a good laugh when I pointed out that the witnesses were named “Tom and Jerry.” I also discovered, as a minister officiating at a wedding, it’s easier to pick up women then you would think. The date went poorly and my friends later told me that had they known who I met, they would have warned me not to. (Amongst other things, on our second date, not only did I find out that she was hopped up on little pink pills her former boyfriend had brought her from Mexico, she asked me to convert to Judaism because she had recently converted and couldn’t date outside of her faith. She was also an atheist. I declined a third date).  That couple also went their separate ways, so my first two weddings, one as groom and one as minister, ended in divorce.  Why I Stopped Officiating Weddings  The third wedding made me swear off officiating at weddings ever again. It was a favor for a friend of a friend. A couple couldn’t afford much, so they wanted a wedding in their home. I agreed, and spoke with the couple on the phone. I asked them about the theme (“non-religious”), who they’d like to include (“uh, we’re not sure”), if they’d be exchanging rings (“we don’t know”), and when they could send me a copy of their vows (“can you write them, please?“). Not one answer was particularly bad, but all of them together, coupled with very strange vibes from the phone call told me that something was, well, off.  I wrote their vows and called them back to arrange a rehearsal. They were busy, they said. Every time I called to try to arrange a rehearsal, they had an excuse. I simply could not get a rehearsal with them. So I finally said I’d show up to the wedding early, meet with them, and have a quick run through of the vows. In turn, I shortened the vows, knowing this was going poorly. It’s a very good thing that I did.  The day of the wedding, my friend (er, we’ll call him Charlie), who asked me if I would help the couple out, drove me to their place in the middle of nowhere. It was early October and they had decided on a Halloween-themed wedding. As we pulled up the house, little white trash bags hung from the trees. I later realized they were supposed to be ghosts, but they looked for all the world like Klan hoods.  As I walked inside, I immediately assumed a “minister” persona, wanting to put the people at ease. Charlie introduced me to sexy vampires, unshaved cowboys, and one under-age Pocahantas   who wore an ill-fitting costume and no bra. She was flashing everyone but no seemed bothered. I bit my tongue as I smiled. The phrase “feeling like a nun in a brothel” never felt more appropriate.  I was ushered into the back to meet the bride and while she stood their, half-naked, she read through the vows quickly and handed them back to me, assuring me they were fine. I then met the groom in a back bedroom and he told me he was busy and had no time to read the vows. I was brushed off quickly. It felt weird. No matter. I had shortened the vows because they wouldn’t meet for a rehearsal, but this was still very unsettling.  Finally the ceremony started, the happy couple faced me, their assembled family and friends faced me, I handed each of them a short copy of the vows they would read, and I started.  “Dearly beloved, we are gathered here today to celebrate the marriage of ...”  And that, dear reader, is when I realized that in the two months of failing to arrange a rehearsal, I had no fucking idea what their names were.  “... of the lovely bride and her handsome groom.”  The audience smiled. I had gotten away with it.  I then turned to the bride and asked her to read her section of the vows. I was proud of the non-religious, life-affirming, gender-equal vows I had written. None of this, “honor and obey” nonsense. She read the vows and the audience smiled again.  I then turned to the groom and asked him to read his section of the vows. He stumbled and I discreetly pointed to the section clearly labeled “groom” so he could recover.  That was when I discovered that the groom was functionally illiterate. He couldn’t read the vows. He slowly, awkwardly, sounded out the words and I held a supporting smile as I died inside. He was probably dying, too. As was his fiancée. This was a low-point in my life.  After the groom finished, I quickly got through the “do you take your bride ...” and “do you take your groom ...“—still omitting the names I didn’t know—and finished with, “you may now kiss your bride.”  After the kiss finished and the applause died down, I had them turn to face the audience and said, “I now introduce you to the happily married couple!”  More applause, followed by dead silence due to our lack of rehearsal. I had no idea what to do next. The audience stared at us. I stared back. I silently cursed Charlie (sorry, Charlie). Then I piped up with, “how about a question and answer session? I’m sure we’d all like to what their life plans are.”  That’s me, the professional speaker. Always prepared to improvise.  After another awkward pause, the audience got into the swing of things and started peppering the bride and groom with questions as I embarrassedly slunk away to the kitchen, desperately in need of a drink. That’s when I discovered it was a “bring your own beer” wedding and no one had bothered to tell me. I was filling a glass with water when some kind soul hurried up and pressed a can of room temperature Olympia beer into my hand. Olympia beer must always be served ice cold because you desperately want dead taste buds when you drink it, but I was grateful.  Later, Charlie and I stood on the back porch, smoking, drinking, and rather uncharitably speculating about how long the marriage was going to last, when the back door slammed open. An angel with a cigarette dangling from her lips burst out, cussing up a storm. Slipping back into “minister” mode, I asked, “what’s bothering you, child?”  “My sister married an asshole!”  I told Charlie I was never doing another wedding. He was, and still is, a friend, but I can never quite forgive that nightmare.  A couple of years later, I learned that the smoking angel was being charitable. The marriage ended very badly.  Why I Started Officiating Weddings Again  But that’s only four weddings. Given that I had sworn off weddings, it’s probably a surprise that I officiated at seventeen more. Over the course of two days.  In March of 2004, Multnomah County of Oregon briefly legalized same-sex weddings  , a full two months before Massachusetts started issuing marriage licenses to same-sex couples  . In fact, the political mess was so awful that at one point, Bend, Oregon banned all marriages  , whether they were same-sex or not.  The Multnomah County ruling was a contentious affair, with the city councilors passing the ruling on a day that they knew an objecting councilor would not be present. County clerks were instructed to waive the three-day waiting period and the 60 license applications a day soared to 400.  Given the overwhelming numbers and the fact that many ministers would not officiate, there was a call for ministers to perform the ceremonies. On March 4, 2004, I called up a woman I was due to have a blind date with the next day and informed her I would be cancelling it because I had to help. She was delighted with my excuse (and we later started dating).  On March 5th, I found myself standing on the sidewalk outside the Multnomah County Courthouse, wearing a nice suit with a clerical collar. I had bought the collar years ago for a Halloween party and was happy to discover that it was a real collar from a ministerial supply company.    The Multnomah County Courthouse Source    One couple I married had been together for eight years and another for seventeen. Their joy at finally being married was wonderful. They cried, they laughed, they hugged and kissed. It was a great thing. I can’t understand how anyone could tell them “no.”  Only one couple gave me pause, a couple of boys who looked eighteen. I wasn’t bothered by the fact that they were gay, of course. It was just that they were so young and I was worried they weren’t mature enough to be married, much less face the discrimination that they were going to face, as evidenced by people standing around with signs announcing that we were going to burn in hell. However, their mothers were with them, beaming with pride. Of course I officiated.  As for myself, I should have brought a lot more water with me. My throat was getting scratchy. I also should have brought some Advil because I found that standing on the sidewalk officiating at wedding after wedding was painful. I was cold and was getting a bit rummy at the end. I stared at some of the forms and once had trouble figuring out what my name was over the shouting of protestors.  There were loud cheers for the first wedding, trickling down to a handful near the end of the day. I married seven couples that day and it was awkard at times. I would ask their names and carefully write them down on their copy of the vows I had hastily written the night before. I had already learned my lesson about forgetting names. It was also mortifying to ask, “For purposes of filling out the paperwork, who will be the groom?” Unsurprisingly, the paperwork hadn’t been updated for same-sex couples.  I also needed two witnesses for every wedding and some couples were there alone, no family or friends to support them. I had no trouble finding volunteers in the crowd.  When the day ended, I was approached by another minister who was also dead tired. He explained that he had secured a free banquet room for the following day and had arranged to have couples sent there for their ceremonies. He saw my ceremonies and asked if I could come help. I was delighted to.  Somewhere out there is news footage from a local television station filming me doing weddings on the sidewalks. I had several calls from excited friends who congratulated me. I wish I could find that footage.   The next day I showed up at the restaurant and throughout the course of the day, the other minister and I would trade off batches of weddings. I performed ten of them that day. Most were lovely and the couples were often crying at the end. One was horrendous when one of the men was very belligerent, demanding to know why I was wearing a clerical collar. I pointed out that I was doing him a favor and his partner quickly calmed him down, but the man glared at me throughout the ceremony. I was tempted to stop and hand them off to the other minister, but his partner looked so embarrassed that I continued.  While waiting for my next batch of weddings, I found myself talking to a delightful lady who was terribly excited that she could finally get married to the love of her life. We had a great conversation and when it was her turn to get married, she asked the other minister if I could officiate instead. It was a lovely ceremony and after I finished, she insisted I come to their reception. I was happy to accept, and she gave me the address of a place nearby that I had never heard of. As I looked at the address, I noticed she included her phone number, “in case I got lost.” I pocketed the address as she was explaining to me that she was actually bisexual and she and her partner had an “open” relationship. And then she just looked at me and smiled.  Oh, shit.  I might be open-minded, but I was not going there. However, I had already agreed, in front of the wedding party, to attend the reception. I made my excuses, found the address, and walked in the front door of a smoke-filled dive bar. Several patrons glanced over, trying to figure out why a minister would be standing there. I had forgotten to remove my collar.  Screw it. I stumbled up to the bar, lit a cigarette and ordered a whisky. This was Portland and the bartender didn’t bat an eyelash. The wedding party arrived, were having a great time, and as soon as I felt I could escape, I made my excuses and left. I never did call that phone number.  The BBC  So those were my twenty-one weddings, only one of which survived. A little over one month later, the Oregon Supreme Court annulled the almost 3,000 same-sex weddings  on the grounds that only the state legislature has the power to regulate marriage. Once again, right and wrong only had a passing flirtation with legal and illegal. My seventeen same-sex marriages were over. I was crushed, but that’s nothing compared to how many lives were devastated by that ruling.  And that is how, in 2010, I found myself working at the BBC in London, trying to deal with a rather awkward phone call.  I was moving in with Leïla, my then-fiancée. She had offered to pack my flat in Pimlico and then we’d move all of the boxes to her flat in Finsbury Park. Leïla is methodical. Organized. Careful. She went through all of my belongings, very carefully, and sorted them into appropriate boxes. She does this every time we move to ensure that when we unpack, nothing is misplaced.  As she was packing my belongings, she stumbled across some paperwork. She’s French and at the time, her English was good, but not fluent. My new fiancée had called me at work, quite unhappy, to know why she found seventeen wedding certificates with my name on them. Was I a bigamist?  We still laugh about that to this day.  Update : I’ve since been reminded of another wedding I officiated at. Oops. That makes 22 of them. That marriage has also ended, so excluding my current marriage, I am proud of say I have a 100% track record of having presiding over failed marriages.  ","title":"My 21 Weddings","url":"\/blog\/my-21-weddings.html"},{"body":"  Obviously, the title is hyperbolic. Not all black people hate white people’s dreadlocks, cornrows, or other stereotypically black hairstyles, but it’s trivial to hit your favorite search engine and read why black people get mad when white people have dreadlocks  . This is a non-issue for many black people, but for others, they’re extremely passionate about it. Why?    Is this cultural appropriation? Source    A couple of years ago, I wrote elsewhere about this topic, but from a slightly different point of view. When white people strongly reject black culture, that’s racism. But when we admire parts of black culture and try to incorporate them, it’s cultural appropriation! Damned if we do, damned if we don’t. Worse, pointing out that some of the earliest depictions of dreadlocks are amost 6,000 years old, from the Minoan civilization in modern-day Greece  can inflame the argument. Making that point often seems tone deaf to the issues the black person is concerned about.  Yet, that’s more or less the argument I made (this is from memory; details are probably off). While my (mostly white) friends generally seemed supportive of this point of view, a black friend of mine said something to the effect of “that’s what a white person would say.” They never responded when I asked what they meant. I really wanted to understand, but it didn’t happen that day.  Today, the Black Lives Matter movement has inserted itself strongly in US politics . Discussions of institutional racism, defunding the police, and similar matters are pervading the American consciousness again, in a way that perhaps wasn’t there since the Civil Rights Movement  . Unsurprisingly, these topics are still terribly divisive. For some, these discussions are cause for them to double-down on their insistence that these issues are overblown. For others—myself included—these discussions have led down a rabbit hole of learning.    My wife’s hair, like my wife, is beautiful.   Unfortunately, the wording of some of these topics makes it even more confusing. For example, in one video when Ben Shapiro is asked about “institional racism”, he immediately demanded to know which [institution] is the racist one?  . It’s a brilliant debating tactic, even if it is fundamentally dishonest. Institutional racism isn’t “that organization down the street hates black people.” Hell, it’s not even racism in the sense that most people think of it. Institutional racism is the result of fundamentally unequal outcomes experienced by different ethnic groups due to structural issues in society’s laws and culture. But that doesn’t roll off the tongue as easily as “institutional racism,” so people like Ben Shapiro nitpick the latter term and hope you won’t notice the bigger picture.    And don’t even get me started on the mess of the slogan “Defund the Police.” It was never about defunding the police and of course it gives many people the idea that we’re arguing for anarchy. For those who are disinclined to listen in the first place, messages like “defund the police” drive us apart instead of together. Even former president Barack Obama has warned against the phrase  , to little effect.  Yet, in the face of the protests and the issues surrounding it, hair seems like a minor issue, right? Well, not it’s not minor for many black people. Chris Rock produced an entire documentary about the black experience with hair.     Simply put, black hair is often different. It’s often tightly curled and considerably more difficult to brush and style than white hair. There’s a culture around black hair and it’s strongly associated with black cultural identity. In 2014, when the US Army banned female soldiers from wearing cornrows, braids, twists and dreadlocks  , they didn’t mention race, but it was clear it was about race. With the limited options that black people have to style their hair, the US Army decided to take those options away. Or there was the case of Vanessa VanDyke. Her school threatened to expel her because of her hair  . It was natural. There were no cornrows, braids, twists, or dreadlocks. It was simply black hair.  Many black people can tell you stories about being told their hair is dirty, or unprofessional, or offensive. People use black hair as an excuse for discrimination. Black hair is “ugly.” And yet when white people wear traditionally back hairstyles, they’re “hip” or “cool.” Usually the grief they get seems to be from black people, not white. Why should white people get away with something black people aren’t allowed to get away with?  I still can’t say that I know what the right answer is here, but the wrong answer is to fail to understand why hair can be such a sensitive topic to black people. I had no idea.  But I think it’s even better summed up in this comment on Reddit  .   There is a long history in the US of black folks being othered for their hairstyles. Braids, locs, natural afros, super tight curls, all these things have set black Americans apart from white Americans. Even though protective hairstyles like braids and locs are necessary to keep many black hair textures healthy, the look was different than white people and therefore “unprofessional” and unattractive. Only in the last few decades have we begun to undo this conditioning. If you want a way better and way more compelling explanation than mine, Chris Rock made a documentary called “Good Hair” to explain black American culture around hair and it’s fascinating.  Anyway, if you’re from a group who has been historically criticized for the way your hair grows out of your head and then see some white people being able to do the same thing with relatively no scrutiny, you’d be pissed too. Cultural appropriation is not simply borrowing things from other cultures, it’s borrowing them without acknowledging what it means to that culture. In the US, black othering and the existence of “good hair (white hair)” is much more relevant to our culture than Viking matting.   ","title":"Why Black People Hate Your Dreads","url":"\/blog\/why-black-people-hate-your-dreads.html"},{"body":"    George Floyd did not deserve to die. Source    Strange fruit on red lines cross blue lines and die.  In the last moments of George Floyd’s life, he was lying on the ground, a terribly symbolic knee on his neck, begging for his life and crying for his mother. It was heartbreaking.  And it was broadcast across the world.  For those who are not Americans, you might wonder how the death of this man has led to riots, has led to the President of the United States threatening to shoot those responsible for “looting” (A threat which, if carried out, violates the oath of office Trump swore to uphold  ).  But it’s shouldn’t be hard to understand. Colin Kaepernick lost his career and was vilified in the press for respectfully “taking a knee” during the National Anthem.   Yet another uppity black person who didn’t know his place. You can’t protest peacefully. You can’t vote your way to equality if enough people vote against you. You can’t protest racism without being accused of being racist  . What avenues are for protest are left?  Strange fruit on red lines cross blue lines and die.  Strange Fruit  , started as a poem and later became that magnificent song by Billie Holiday singing about black people being lynched in the south and hanging from trees (like fruit). Listen to Jill Scott singing Strange Fruit.     Redlining  , of course, is the historical US practice of keeping black people out of white neighborhoods.  The blue line, of course, represents the police.  In a recent study of US police use of force  , we find that black people are two and a half times more likely to be killed by police than white people. The reasons are debated  , but it’s a moot point. As The Guardian found in their comprehensive research of US police killings, US police routinely kill more people in days than other countries do in years  . From that article:   In the first 24 days of 2015, police in the US fatally shot more people than police did in England and Wales, combined, over the past 24 years.   You can argue all you want about why police are killing more black people than white people, but this neatly sidesteps the issue of why US police are killing at all.  Despite that, I’ll sidestep that issue. I’ll even sidestep the issues of racial justice because there should be no need to belabor the point that being black in America is fraught with danger and discrimination. Instead, I want to point out where this could be leading.  The New Civil Rights Movement  The Civil Rights Movement  of the 50s and 60s was people of all skin tones working together to end the widespread discrimination against blacks. The 14th Amendment  guaranteed all US citizens “equal protection of the laws” and the 15th Amendment  gave black people the right to vote.    Thomas Mundy Peterson was the first black person to vote after the passage of the 15th amendment. Source    After the 15th Amendment was passed, Thomas Peterson was the first black person to vote in the United States. The people in his town later raised money and gave him a gold medal celebrating this event. However, Jim Crow laws  , mostly enacted in the Southern US, were passed with the goal of taking away this right.  These laws were largely successful. Redlining ensured that black people were not allowed to live in “white” communities and would be denied the economic opportunities that white people enjoyed. Education systems funded by property taxes meant that white children received better education than black children, which, in turn, helped to ensure that black people in the US would be economically disadvantaged.  But all of that pales in comparison to the thousands of black people who were lynched in the (mostly Southern) US  , almost universally without legal repercussions. According to the NAACP  :   From 1882-1968, 4,743 lynchings occurred in the United States. Of these people that were lynched 3,446 were black. The blacks lynched accounted for 72.7% of the people lynched. These numbers seem large, but it is known that not all of the lynchings were ever recorded. Out of the 4,743 people lynched only 1,297 white people were lynched. That is only 27.3%. Many of the whites lynched were lynched for helping the black or being anti lynching and even for domestic crimes.     The Lynching of Tom Shipp and Abe Smith at Marion, Indiana, August 7, 1930 Source    These and many other issues helped drive the Civil Rights Movement, but today, many Americans don’t realize just how bloody that movement was. Sure, we know about the assassinations of Malcom X and Martin Luther King, Jr. And maybe you’ve a vague recollection of someone named Medgar Evers  , but did you also know about George W. Lee  , Lamar Smith  , Dr. Thomas Brewer  , Herbert Lee  , William Moore  , Louis Allen  , James Chaney  , Andrew Goodman  , Michael Schwerner  , James Reeb  , Viola Liuzzo  , Jonathan Daniels  , Sammy Younge Jr.  , Vernon Dahmer  , Robert W. Spike  , or Wharlest Jackson  ?  No, you probably didn’t know most of those names (I certainly didn’t). The US school system largely omits the history of the Civil Rights Movement  and to this day, it’s hard to get a full accounting of how many people died as a result of that movement, but in reading their stories, one thing becomes clear: many of them knew they might die, but they did what they did anyway. But we usually didn’t learn that in schools; it’s just not taught.  It’s a crime that we’re not taught about all of those people murdered for the crime of thinking that discrimination against black people is wrong.  These issues drove the Civil Rights Movement of the 50s and 60s, but George Floyd’s death, or the future death of another person like him, will drive the new Civil Rights Movement.  Thugs  I recently shared a video on Facebook that went viral  and currently has over 20 million views. I was flooded with friend requests. So I decided to conduct an experiment. With the exception of those accounts who clearly appeared to be bots, I accepted all of those friend requests in an attempt to positively engage with those who had different worldviews from mine. My wife wasn’t happy about this and, I must admit, she was right. I learned quite a bit from that experiment, but the main takeaway was simply that Facebook isn’t the place to have nuanced discussions about political issues.  The latest trainwreck of conversation was by someone reading one of my posts about George Floyd’s death and the subsequent riots and simply replying “Oh well thugs are thugs after all.” Their subsequent comments flowed into anti-LGBT territory and a defence of Trump, just in case anyone misunderstood where their sympathies lay. These attitudes by those who ”friended” me on Facebook are very, very common.  Black people aren’t protesting against the death of George Floyd. They’re just thugs. And what are these thugs protesting against?  Christian Cooper (who happens to be black), a board member of the Audobon Society in New York, was bird watching when he asked a woman to leash her dog in an area where dogs weren’t allowed off leash. Her response (on camera) was to yell “I’m going to tell them there’s an African American man threatening my life!”  Ahmaud Arbery, an unarmed black man out for a jog, was hunted down and killed by three white men in Georgia.  Trayvon Martin, a black teenager, was shot just for walking home.  Botham Shem Jean, also black, was relaxing in his own apartment when he was murdered by an off-duty police officer.  Oscar Grant (coincidentally, black), was held down by police and shot in the back.  In fact, one study of Deaths Due to Use of Lethal Force by Law Enforcement  found:   Victims were ... disproportionately black with a fatality rate 2.8 times higher among blacks than whites ... [Black] victims were more likely to be unarmed than white or Hispanic victims.   Black children have been shot and killed by police for carrying toy guns or cell phones. Eric Garner was choked to death by police  for selling cigarettes. In 2015, an average of two unarmed black people were killed each week by police in the United States  .  A new Civil Rights Movement is starting in the United States.  It’s not about disenfranchisement.  It’s not about redlining.  It’s not about loss of job opportunities.  It’s about being murdered for being black.    Black Lives Matter   There’s a new civil rights movement brewing in the United States of America and it’s long overdue.  ","title":"The Murder of George Floyd and The New Civil Rights Movement","url":"\/blog\/the-murder-of-george-floyd-and-the-new-civil-rights-movement.html"},{"body":"  I’ve been wanting to get back to playing around with AI life simulation. Years ago I was writing a small simulation in Perl. I was going to “evolve” tigers, cows, and grass in a simple food-chain simulation.  The tigers would learn to hunt cows, the cows would learn to eat grass and avoid tigers, and the grass, well, the grass would grow.  I wrote a lot of code. The brains were simple “winner take all” neural networks and the animals could eat, move forward, or turn.  What sort of amazing strategies would these animals evolve? After they evolved, what sort of new animals and constraints could I add to make the simulation more interesting?  I had written plenty of code, lots of tests, and was quite proud of my work. Except it didn’t work.  The cows wandered around aimlessly. The tigers wandered around aimlessly. The grass grew. My animals would not evolve.  I wrote more tests. Maybe they couldn’t see? No, they could see. Maybe eating wasn’t giving them enough energy? No, eating was giving them enough energy.  Maybe their brains didn’t work? No, their brains worked just fine. Mine, however, didn’t. I was stumped.  I kept poking and prodding and theorizing and moaning and finally had an epiphany. I had, once again, written unit tests for everything.  I hadn’t written integration tests. Amongst other things, I teach software testing and over and over again, I warn people how critical integration tests are. But here I was, making the same mistake.  So I wrote integration tests. And that’s when I discovered it.  My animals could think. My animals could see. But I never connected their eyes to their brain. They never thought about what they were seeing.  I had written a voter simulator.  ","title":"My AI Life Simulator","url":"\/blog\/my-ai-life-simulator.html"},{"body":"    Corinna’s First (mis)Steps  Good OO Design  My Mistake  Politics  Class Data and Methods  Inclusion  Semantics  Syntax  State Variables  The Future  Conclusion  Update     Corinna’s First (mis)Steps  Note: Corinna is the name of Ovid’s love interest in the Amores  .   I had been working on the design for Corinna, a proposed object system for the Perl core  , for several months, inspired by the work of Stevan Little  , pulling ideas from many other sources, including far too much reading about effective object-oriented design. Sawyer X  , then the Perl pumpking (a now defunct role, but it was the person overseeing Perl language development), urged me to stop working on the implementation and just focus on the design. Design something great and he’d find someone to implement it.  When I first announced Corinna (then known as “Cor”), it was met with enthusiasm, indifference, and hostility, depending on whom you asked. In fact, I was convinced that what I had proposed was solid enough that I was sure the community would buy into it easily. They did not.  Truth be told, they were right to not get excited. Ignoring the fact that people have wanted to get an object system in the Perl core for years and no one had succeeded, the fact is that Corinna wasn’t really that interesting in terms of an OO system for the Perl community. I had spent so much time researching good OO practices that I hadn’t take into consideration the impact of decades of ingrained behaviors in the Perl community. Here’s what an early version looked like.  class Cache::LRU v0.01 { use Hash::Ordered; has cache =&gt; ( default =&gt; method { Hash::Ordered-&gt;new } ); has max_size =&gt; ( default =&gt; method { 20 } ); method set ( $key, $value ) { if ( self-&gt;cache-&gt;exists($key) ) { self-&gt;cache-&gt;delete($key); } elsif ( self-&gt;cache-&gt;keys &gt; self-&gt;max_size ) { self-&gt;cache-&gt;shift; } self-&gt;cache-&gt;set( $key, $value ); } method get ($key) { if ( $self-&gt;exists($key) ) { my $value = $self-&gt;cache-&gt;get($key); $self-&gt;set( $key, $value ); # put it at the front return $value; } return; } }  I won’t bore you with the details of the bad decisions I made, but you can read the initial Corinna specification here  . Let’s look at one mistake I made in that specification:   Note: “slots” are internal data for the object. They provide no public API. By not defining standard is =&gt; 'ro' , is =&gt; 'rw' , etc., we avoid the trap of making it natural to expose everything. Instead, just a little extra work is needed by the developer to wrap slots with methods, thereby providing an affordance to keep the public interface smaller (which is generally accepted as good OO practice).   So you couldn’t call $self-&gt;cache or $self-&gt;max_size outside of the class.  Good OO Design  So I said encapsulation is “generally accepted as good OO practice.”  Yes, that’s true. And if you’re very familiar with OO design best practices, you might even agree. Java developers learned a long time ago not to make their object attributes public, but Perl, frankly, didn’t really have simple ways of enforcing encapsulation. In fact, in the early days of OO prorgamming in Perl, you’d often see an object constructed like this:  package SomeClass; sub new { my ( $class, $name ) = @_; bless { name =&gt; $name, _items =&gt; [], }, $class; } 1;  With that, even outside the class you could inspect data with $object-&gt;{name} and $object-&gt;{_items} . But why the leading underscore on _items ? Because that was a signal to other developers that _items was meant to be private. It was asking them to not touch that because Perl didn’t offer an easy way to ensure encapsulation.  Something similar for Moo\/se could be written like this:  package SomeClass; use Moose; has &#39;name&#39; =&gt; ( is =&gt; &#39;ro&#39;, isa =&gt; &#39;Str, required =&gt; 1, ); has &#39;_items&#39; =&gt; ( is =&gt; &#39;ro&#39;, isa =&gt; &#39;ArrayRef&#39;, default =&gt; sub {[]}, init_arg =&gt; undef, ); 1;  That almost the same thing, but now you have a reader for $object-&gt;_items and you still don’t get encapsulation. Bad, right?  Well, yeah. That’s bad. All Around the World  has repeatedly gone into clients and fixed broken code which is broken for no other reason than not respecting encapsulation (drop me a line if you’d like to hire us).  There have been attempts to introduce “inside-out” objects  which properly encapsulate their data, but these have never caught on. They are awkward to write and, most importantly, you can’t just call Dumper($object) to see its internal state while debugging. That alone may have killed them.  My Mistake  So if encapsulation is good, and violating encapsulation is bad, why were people upset with my proposal?  Amongst other things, if I wanted to expose the max_size slot directly, I had to write a helper method:  class Cache::LRU v0.01 { use Hash::Ordered; has cache =&gt; ( default =&gt; method { Hash::Ordered-&gt;new } ); has max_size =&gt; ( default =&gt; method { 20 } ); method max_size () { return self-&gt;max_size } ... }  If I wanted people to be able to change the max size value:  method max_size ($new_size) { if (@_ &gt; 1) { # guaranteed to confuse new developers self-&gt;max_size($new_size); } else { return self-&gt;max_size; } }  And there are many, many ways to write the above and get it wrong. Corinna is supposed to make it easier to focus on writing good OO code. Here, it was letting developers fall back to writing bad code that Corinna could easily write. Why should people have to write it? If I offered some convenience here, how should it look? Enough people argued against the design that I realized I made a mistake. Maybe encapsulation is good, but my strictness was over the top. Which leads me to the entire point of this blog entry:  If you’re not effective, it doesn’t matter if you’re right.  Was I right to strive for such encapsulation? I honestly don’t know, but if it meant that no one would use my OO system, it didn’t matter. I had to change something.  Politics  It was pointed out to me that since I had been working in isolation, I hadn’t had the chance to hear, much less incorporate, the views of others. It was time for politics.  Out of all of the online definitions about politics, I really like the opening of Wikipedia  the best:   Politics is the set of activities that are associated with making decisions in groups ...   That’s it. People often hate the word “politics” and say things like “politics has no place in the workplace” (or community, or mailing list, or whatever). But politics is nothing more than helping groups form a consensus. When you have a group as large as the Perl community, if you’re not interested in forming a consensus for a large project, you’ve already put yourself behind the eight ball and that’s what I had done.    Source    So an IRC channel was created (irc.perl.org #cor) and a github project  was started and anyone with an interest in the project was welcome to share their thoughts.  To be honest, this was challenging for me. First, I don’t respond well to profanity or strong language and there were certainly some emotions flying in the early days. Second, nobody likes being told their baby is ugly, and people were telling me that. For example, when I was trying to figure out how to handle class data, here’s part of the conversation (if you leave a comment, I will delete it if it’s identifying this person):   developer: Ovid: this is not an attack on you or at all, I’m just about to rant about class data, ok  developer: class data is bullSHIT  developer: there is no such thing  developer: it’s a stupid-ass excuse in java land for them not having globals  developer: just make it a fucking global  developer: or a fucking constant method  developer: sub foo { ‘value' }  developer: there, it’s a constant method, it’s fine, fuck off   When I see things like that, I tend to tune out. That’s not helpful when I need to understand where people are coming from. And I’ve read that multiple times and I still don’t see the technical argument there (to be fair, this went on for a bit, but I saw nothing explaining why global variables are better than class data. It could just be that I’m a bear of little brain).  I strongly disagree that globals are better than class data because I’ve worked on codebases with plenty of globals and, even if they’re immutable, there’s the lifecycle question of when the come into existence. Plus, class data is clearly associated with a class, so at the very least, when I call my $foo = SomeClass-&gt;data (assuming it’s needed outside the class), the class can maintain responsibility for what that data is. Or if the class data is private to the class (as it often is), a global breaks that entirely. But nonetheless, I have to get buy-in for Corinna and that means listening to people and trying to figure out what the real complaints were and how (and whether) to address them.  This worked well for slot encapsulation because we eventually came up with this:  slot $name :reader :writer;  The $name variable can be read via $object-&gt;name and can be set via $object-&gt;set_name($new_name) . Yes, this tremendously violates encapsulation because Corinna does not yet have a way of enforcing type constraints, but mutability at least isn’t the default. And when we later incorporate type constraints, the external interface of the classes won’t have to change. I got buy-in for the design, at the cost of slightly compromising some of my design goals. I think it’s a good trade off and I think Corinna is better off for this.  But what about class data?  Class Data and Methods  Should we include class data and methods in Corinna? I discovered that for many people, the answer was obvious, but their obvious answers disagreed with other people’s obvious answers. What makes it worse is that this isn’t just a “yes or no” question. We have multiple questions.   Should we include class data and methods?  What should the semantics be?  What should the syntax be?   This isn’t easy. We got past one and two with difficulty, but number three has been a bit of a trainwreck. And I’m the conductor.  Inclusion  I think it was pretty much universally agreed that we had to have class methods. These wouldn’t have $self injected into them, nor would they have any access to instance data. In fact, done properly, we could make violations of that a compile-time failure. That’s a huge win for Perl. They might look like this:  slot $some_data; # instance data common method foo() { # $class is available here # $self and $some_data are not available here }  One use of those is for alternate constructors. In Moo\/se, you have BUILDARGS for fiddling with arguments to new . In Corinna, at least for the MVP, you write an alternate constructor (we don’t have method overloading in Perl).  class Box { slot ( $height, $width, $depth ) :param; slot $volume :reader = $height * $width * $depth; common method new_cube ($length) { return $class-&gt;new( height =&gt; $length, width =&gt; $length, depth =&gt; $length, ); } }  With the above, you can create a cube with Box-&gt;new_cube(3) . No more messing around with BUILDARGS and trying to remember the syntax.  So if we should have class methods, should we have class data? Well, it’s been pointed out that it’s literally impossible to avoid:  class Box { my $num_instances = 0; # class data!! slot ( $height, $width, $depth ) :param; slot $volume :reader = $height * $width * $depth; ADJUST { $num_instances++ } DESTRUCT { $num_instances-- } common method new_cube ($length) { return $class-&gt;new( height =&gt; $length, width =&gt; $length, depth =&gt; $length, ); } common method inventory_count () { $num_instances } }  So at a very practical level, whether or not we want class data, we have it. In fact, even if we omitted class methods, we’d still have class data. So let’s work with what we have.  Semantics  We have a lot of the semantics described here  , but it could use some more work. However, the general idea of behavior isn’t controversial enough that I want to spend too much time here.  Syntax  Now we have a problem. In Moo\/se, we BUILD , DEMOLISH , and has , which have been renamed to ADJUST , DESTRUCT , and slot in Corinna in part because they’re different beasts. We don’t want to have BUILD in Corinna and have Moo\/se developers think it’s the same thing. If we get an analog to BUILDARGS , it will probably be called CONSTRUCT for the same reason.  So one of our design goals is that different things should look different.  Another design goal is that we do not wish to overload the meaning of things. Thus, we agreed that reusing the class keyword ( class method foo() {...} or class slot $foo ) was probably a bad idea (it turns out to be spectacularly bad if we get to inner classes, but let’s not go there yet).  By the same reasoning that “different things should look different,” similar things should look similar. In Java, class data and methods are declared with the static keyword.  public class MyClass { private String name; \/\/ class data public static int numberOfItems; public MyClass(String name) { this.name = name; } \/\/ class method public static void setSomeClassData(int value) { MyClass.numberOfItems = value; } }  A developer can easily understand how the two are analogous. But do we need this for Corinna? Here’s the “accidental” class data we could not avoid.  class Box { my $num_instances = 0; # class data!! slot ( $height, $width, $depth ) :param; slot $volume :reader = $height * $width * $depth; ADJUST { $num_instances++ } DESTRUCT { $num_instances-- } common method new_cube ($length) { return $class-&gt;new( height =&gt; $length, width =&gt; $length, depth =&gt; $length, ); } common method inventory_count () { $num_instances } }  But we could get rid of inventory_count method by supplying a reader (and even renaming it).  class Box { my $num_instances :reader(inventory_count) = 0; # class data!! slot ( $height, $width, $depth ) :param; slot $volume :reader = $height * $width * $depth; ADJUST { $num_instances++ } DESTRUCT { $num_instances-- } common method new_cube ($length) { return $class-&gt;new( height =&gt; $length, width =&gt; $length, depth =&gt; $length, ); } }  So right off the bat, for new developers, we need to teach them when they can and cannot use slot attributes with my variables.  Also, Perl has the conflation of package and class , along with sub and method . Do want to add my for class data and for lexical variables? And as Damian Conway has pointed out  , static analysis tools are already hard enough to write for Perl, given the overloaded meaning of many keywords.  And if we do accept the notion that similar things should look similar, why would class methods and class data have different declarators? We can’t just say my method foo() {...} because that clearly implies it’s a private method.  Or we can adopt the approach other OO languages such as Java, C++, C#, and Swift have done and use a single keyword to explain the same concept: these things are bound to the class and not an instance of the class. For the aforementioned languages, that keyword was static , but it was strongly shot down as “not good for us” due to possible confusion with the state keyword and the fact that different languages sometimes use static to mean different things. Different things should look different.  shared seems good, but that implies threads to many people  , so that was also shot down.  I’m not sure who came up with the word common (it may have been me back in February of 2021, according to IRC logs) and so far it seems like the least-bad alternative. (Another suggestion I proposed at that time was mutual )  However, there are those who are strongly in favor of my , including adding attributes to it—if it’s in Corinna and not inside a method—and strongly object to common on the grounds that all methods defined in a class are common to every instance of that class. They have a point about common being a poor choice, but I don’t have a good one and I suspect that, over time, it won’t even be noticed (I may live to regret typing that).  So while I’m trying to figure all of this out, Damian Conway posted an extensive endorsement of Corinna  . To illustrate one of his points, he shared a class written in Dios, which an OO system for Perl which he introduced in his “Three Little Words” presentation.     He wrote the following class.  use Dios; class Account { state $next_ID = &#39;AAA0001&#39;; has $.name is rw is required; has $.balance = 0; has $.ID = $next_ID++; method deposit ($amount) { $balance += $amount; } method report ($fh = *STDOUT) { $fh-&gt;say( &quot;$ID: $balance&quot; ); } }  That’s almost how you would write it in Corinna, but that’s not what I really noticed.  I kept staring at that state variable he used to declare class data.  Everyone arguing for reusing an existing declarator to declare class data in Corinna was arguing for my . Here’s Damian, using state .  I couldn’t get that out of my mind. And then I started thinking about inner classes, but let’s not go there yet. Let’s talk about state and why this is important.  State Variables  perldoc -f state  says:   state declares a lexically scoped variable, just like my. However, those variables will never be reinitialized, contrary to lexical variables that are reinitialized each time their enclosing block is entered. See “Persistent Private Variables” in perlsub  for details.   What does that mean? Well, first, let’s run the following code.  sub printit { state $this = 1; my $that = 1; $this++; $that++; say &quot;state $this and my $that&quot;; } printit() for 1 .. 3;  That prints out:  state 2 and my 2 state 3 and my 2 state 4 and my 2  As you can see, state variables are like static variables in C. They are declared once and retain their value between calls. They kinda look like static members in Java.  Let’s look at state some more, this time returning an anonymous subroutine with the variables.  sub printit ($name) { return sub { state $this = 1; my $that = 1; $this++; $that++; say &quot;$name: state $this and my $that&quot;; } } my $first = printit(&#39;first&#39;); my $second = printit(&#39;second&#39;); $first-&gt;() for 1 .. 3; $second-&gt;() for 1 .. 3;  And that prints out:  first: state 2 and my 2 first: state 3 and my 2 first: state 4 and my 2 second: state 2 and my 2 second: state 3 and my 2 second: state 4 and my 2  Hmm, perldoc -f state says that state variables are only initialized once, but in the case of returning an anonymous sub, we’ve created a new lexical scope and we have a different state variable.  Just for completeness, let’s define those variables inside the outer sub, but outside the inner sub.  sub printit ($name) { state $this = 1; my $that = 1; return sub { $this++; $that++; say &quot;$name: state $this and my $that&quot;; } } my $first = printit(&#39;first&#39;); my $second = printit(&#39;second&#39;); $first-&gt;() for 1 .. 3; $second-&gt;() for 1 .. 3;  And that prints out:  first: state 2 and my 2 first: state 3 and my 3 first: state 4 and my 4 second: state 5 and my 2 second: state 6 and my 3 second: state 7 and my 4  So, depending on how we declare those variables and what the enclosing scope should be, we get different results. This is more or less as expected, though creating a new lexical scope and having the state variables re-initialized might surprise some because I don’t think it’s clearly documented.  But what does that mean for class data?  Part of my job is to ensure that Corinna doesn’t break existing Perl. However, I need to ensure that Corinna doesn’t hobble future Perl, either. That’s much harder because we can’t predict the future.  The Future  There are two things we would love to see in the future for Perl. One is inner classes and the second is anonymous classes. “Anonymous classes” already feels “Perlish” because we have anonymous subroutines and most Perl developers are familiar with the concept of closures. But let’s discuss inner classes first since many people are not familiar with them. Let’s look at some examples from the Java documentation  .  class OuterClass { ... class InnerClass { ... } static class StaticNestedClass { ... } }  The InnerClass has access to all static (class) and instance variables in OuterClass , while the StaticNestedClass class only has access to the static variables.  What this means is that you can put together a collection of classes and encapsulate the “helper” classes. When people talk about allowing classes to “trust” one another with their data but not share it with the outside world, this is a way to do that while still maintaining encapsulation.  For Corinna, it might look like this:  class OuterClass { ... class InnerClass { ... } common class StaticNestedClass { ... } }  So we’d immediately have something we can reason about, with well-defined, battle-tested semantics from the Java language (if we’re allowed to steal from other languages, Java should be on that list. No language bigotry, please).  (As an aside, this is why we can’t reuse the class keyword for class data and methods. How would we describe a static inner class? class class StaticNestedClass {...} ?)  Next, let’s consider an anonymous class. Here’s one way to think about it.  my $thing = class { slot $foo; slot $name :param; ... }; my $o1 = $thing-&gt;new(name =&gt; $name1); my $o2 = $thing-&gt;new(name =&gt; $name2);  We could go the Java route and allow declaration and instantiation at the same time  , but I don’t think that gains us anything:  my $object = (class { slot $foo; slot $name :param; ... })-&gt;new(name =&gt; $name1);  But consider this:  class OuterClass { ... private class InnerClass { ... method some_method (%args) { return class { state $class_data = 17; # or my $class_data = 17 slot $name :param; }; } } }  So we have a private inner class which returns anonymous metaclass instances with state or my variables being used for class data. Are they shared across all metaclass instances or not? I would think “no”, but someone else might make a reasonable argument otherwise. And should it be state or my ? Do either really connote “global to this class regardless of how the class is generated”?  And what we’re talking about is something speculative, years in the future, where the existing semantics of my or state might be perfectly appropriate. Or after we get there, we might discover that they’re not appropriate and we’ve backed ourselves into a corner because we decided to reuse a handy thing in the language.  Conclusion  There are no good answers here, but I had to make a call. And I decided to err on the side of established conventions in popular languages, and potential safety for the future of Perl. I also didn’t want to potentially overload the meaning of existing syntax.  There are already people who have let me know that they’re very upset with this decision. There are others who are fine with this decision; they just want to get Corinna in core. In this case, I don’t think there’s a “safe” political solution. So I decided to play it safe technically.  People might come back to me later and make a strong argument that I screwed up. That’s fine. I welcome those arguments and I might change my mind, but the arguments have raged since February with no sign of consensus. I had to make a call. I might not have made the right call, but it needed to be done.  Update  I’ve been getting some feedback on this. I’ve now been going back through tickets, emails, and IRC logs and I see that there some agreement on class method foo () {...} , but a lot of disagreement over class $foo . There was some discussion of class slot $foo being OK. There has been so much discussion of this issue for months on end, on IRC, email, and github, that I’ve not remembered all of the fine-grained detail well.  The use of a class as a both a class declarator and a non-access modifier overloads the meaning of class and violates the “different things should look different” principle (to be fair, it’s not the only place we’ve violated this). And this still doesn’t address the awkward case of declaring a static inner class: class class InnerClass {...} .  Looks like this debate is not done.  ","title":"Politics in Programming","url":"\/blog\/politics-in-programming.html"},{"body":"  It goes without saying that COVID-19, better known as the coronavirus, is dominating the news right now. Plenty of people are worried and there’s plenty of “fake news” out there. However, an article entitled The Coronavirus Is Much Worse Than You Think  , by Samuel Paul Veissière Ph.D. and published in Psychology Today, not only has a terrible clickbait title, but commits one of the worst sins a news organization can commit: lying without lying.    Source    While I’m sure that Veissière didn’t mean to do this directly, I’m nonetheless horrified by this article because it’s making the rounds and I want a handy link to share to debunk the nonsense right now.  The article’s premise is actually sound: you’re very unlikely to contract the coronavirus and, even if you do, you’re unlikely to die from it, so there’s no need to panic.  So far, so good. Except the author makes a terrible, terrible mistake by initially omitting the word “now”.  So let’s look at his initial question:   Why then are so many countries implementing quarantine measures, shutting down their borders, schools, and soccer games for something that is less likely to happen to anyone than drowning in a single year, or even being hit by lightning in one’s lifetime? Why is the stock-market crashing, and why are school and workplace mass emails, news headlines, social media feeds, and face-to-face conversations dominated by stories about what is essentially a new strand of mild to moderate flu?   The error in that should be blatantly obvious, but it would be hilarious (were the situation not so grave) given that the author followed this up with:   To understand this strange dynamic, consider people’s blatant inability to make statistically correct inferences about actual risk ...   OK, let’s talk about statistics. In particular, we’re going to talk about “dependent” and “independent” events.  What are the chances that I contract coronavirus? Right now, they’re pretty small. But what if my wife catches coronavirus? My chances increase dramatically. Thus, the chances that I’ll contract coronavirus are highly dependent on whether or not those around me catch coronavirus. Thus, these are what we call “dependent events” and the chances of one materially effect the chances of the other.  This is what Samuel Paul Veissière, Ph.D., doesn’t seem to understand about statistics. To answer his original question about the quarantine when the risk is so low: the risk can remain low if and only if the quarantine works.  Asshole. (Dr. Veissière is probably actually a wonderful person, but this sort of irresponsible writing really makes me mad).  So the risk is dependent on the quarantine, but let’s talk about independent events because they make the situation worse. My wife and I have a young daughter. Let’s say that one of us catches the disease. The chances are high that all of us will then catch the disease. But the chances of any of us dying from the disease are independent of one another. But what does that mean?  Currently, it appears that the chances of our daughter dying are pretty close to zero. The chances of my wife dying appear to be around 2%, and the chances of my dying are around 4%.   . Well, it turns out that if you do the math , that works out to a 5.9% chance of one of us dying. Instead of a 1 in 50 chance of losing a parent, our daughter has a 1 in 17 chance of losing a parent.  So yeah, coronavirus is a big deal and not something for otherwise respectable publications like Psychology Today to brush off.  But what happens globally? According to Prof. Marc Lipsitch, Professor. of Epidemiology, Harvard School of Public Health, up to 40% to 70% of the world population could get infected.  . He also has very good historical data to back him up. It doesn’t mean this will happen, but if we don’t take preventive measures, it could happen. So what does that mean?  If we’re generous and assume the lower number of 40% of the population is infected, and we assume a 2% (overall) mortality rate, that’s around sixty million dead, or about 170K per day. (To be honest, there’s some hand-waving in this due to unknown rates of infection, re-infection, and so on).  It’s estimated that 150K people die every day (globally), so we’ve more than doubled the number of people dead. But these aren’t people being hit by cars, bullets (go USA!), or dropping dead from a heart attack or stroke. These are people whose deaths are painful and take some time. We’re already overwhelming some of the public health systems. We’re already shutting down businesses all over the world. And that’s long before pandemic status. If we don’t contain the virus, you can expect fewer food deliveries, fewer medical deliveries, more companies failing, and so on. And if you’re one of the unlucky people to contract the virus, you know that probability of dying is significantly higher than it’s ever been.  Oh, and about that 2% chance? While the Dr. Veissière blithely reassures us that we’re all being silly, let’s talk about comorbidities. These are other medical conditions you might have at the same time. In particular, cardiovascular disease, diabetes, high blood pressure, chronic respiratory infections, and other issues appear to dramatically increase your chance of dying from coronavirus.  As a final thought, just in case you think you’re safe, Dr. Veissière does have this to say:   The bad news for you is that, if you live in a densely populated area, you are very likely to contract the coronavirus — if not this year, next year, or the year after as it undergoes its seasonal global migration pattern with its zoonotic cousins.   So there you go. Buried at the end of his article, after he seems to discourage us from quarantines, after he presents a bunch of evolutionary history which isn’t relevant to most people’s concerns, he finally admits that most of us are going to get the disease sooner or later.  COVID-19 certainly isn’t the end of humanity, it’s a very serious problem that absolutely doesn’t deserve this type of sloppy reporting.  Update : it turns out that scientists are also calling BS on this drivel.  Seems that not only was the statistical premise wrong, but the biology was also dead wrong.  ","title":"Coronavirus Fake News","url":"\/blog\/coronavirus-fake-news.html"},{"body":"  Space Exploration Technologies Corp., commonly known as SpaceX, the darling of the launch industry, is one of the most valuable private companies in the US. With almost quarter of all launches worldwide being handled by SpaceX, and with Starship set to revolutionize the launch industry , SpaceX appears to be an unstoppable industry force, but is it?  Eric Berger, the senior space editor at Ars Technica  , posted a bombshell of a tweet.   After the briefing, Administrator Bill Nelson told me that he spoke with SpaceX&#39;s Gwynne Shotwell last weekend. He asked if Twitter\/Musk was going to be a distraction to SpaceX and its commitments to NASA. &quot;She said to me in no uncertain terms, &#39;I assure you it is not.&#39;&quot; &mdash; Eric Berger (@SciGuySpace) December 11, 2022   To put this into context, back in 2019, the Department of Defense (DoD) was investigating whether or not it properly certified SpaceX to provide national security launches  . We do not know why this happened.  Several weeks later, Elon Musk’s security clearance was being reviewed  because he was smoking marijuana on Joe Rogan’s podcast. Even though that was legal under California law, it’s still a Class A misdemeanor under federal law. In fact, three or more possession convictions  can raise this to a felony and this makes it much harder to get a security clearance. If he loses it, he may lose access to the classified material necessary to plan launches for the DoD. That doesn’t mean SpaceX is out, but it would make it harder for him to have an active role.  Then, in October of this year, reports surfaced about Biden administration concerns over the national security implications of Musk’s behavior  . He’s made pro-Russian statements, his reliance on foreign investors to buy Twitter (including Saudi Arabia, Qatar, and a cryptocurrency exchange run by a Chinese-Canadian), and his hints of cutting off Starlink service in Ukraine have all combined to send very confusing signals as to his intentions. The White House later denied those reports of having concerns  .  Shortly after this, SpaceX had a major management reshuffle  , with COO Gwynne Shotwell and vice president Mark Juncosa being brought in to focus on Starship production. Elon Musk is famous for personally handling things like this, but seems to be bogged down by his Twitter antics.  The DoD is in a bind. United Launch Alliance (ULA) are replacing the Atlas 5 rocket with the Vulcan, but Blue Origin has struggled to produce the BE-4 engines needed . Just over a month ago, Blue Origin  delivered the first BE-4 engines, but they’re years late and there’s no evidence that Blue Origin can ramp up production to match SpaceX. Now that Soyuz rockets are no longer an option, the DoD faces the nightmare of SpaceX or nothing. There are many other launch companies, including Firefly Aerospace and Rocket Lab, but they can’t match the cadence, reliability, or launch capabilities of SpaceX. Other rocket companies are either non-US based, or are little more than PowerPoint presentations.  Now the situation is getting worse. The industry needs more launches than launch providers can supply.  Even if Musk ultimately is determined to be a security risk, there’s no viable alternative to SpaceX other than ULA. If Blue Origin can’t deliver enough of the BE-4 engines, it’s SpaceX or nothing. This is not a good position to be in.  This is why Berger’s tweet about NASA is so important. NASA is asking, point-blank, if they can trust Musk. The DoD has had concerns. The Biden administration has (allegedly, but probably true) concerns, and now NASA is asking. So much of the industry is relying on SpaceX and Musk is busy tweeting things like my pronouns are Prosecute\/Fauci  . Not only is this buying into far-right conspiracy theories—something which is sure to put people on edge—but it’s mocking the LGBT community. The latter point probably isn’t an issue from a security point of view, but the QAnon-style tweets, his comments in support of Russia, his support for white supremacists and neo-Nazis on Twitter  are sounding alarm bells.  After months of chaos, Musk seems be begrudgingly admitting that his tenure as head of Twitter may not have been a good thing. He’s let Twitter users vote on whether he runs Twitter and they clearly voted him out.   Should I step down as head of Twitter? I will abide by the results of this poll. &mdash; Elon Musk (@elonmusk) December 18, 2022    Perhaps Musk getting booed off stage  is the wakeup call he needed. Perhaps he couldn’t handle the constant hate thrown at him. Or perhaps he’s realized that running a social media company is harder than he thought.  Whatever the reason, if Musk does step down, it might not happen for months and this won’t change much. He will still own Twitter and whoever takes over his role will be doing what Musk tells him to do. But maybe this will end the daily drama that has accompanied Musk’s takeover of Twitter. This might also allow Musk to give much needed attention to SpaceX.  But I can’t help but shake the feeling that SpaceX might not want that attention.  ","title":"Is SpaceX Stumbling?","url":"\/blog\/is-spacex-stumbling.html"},{"body":"","title":"Why You Will Say \"No\" to Living Abroad","url":"\/blog\/why-you-will-stay-no-to-living-abroad.html"},{"body":"  I always forget many things about basic statistics, but I find I’m more likely to remember them if I write about them. Hence, this blog post. It’s written to help me remember something I keep forgetting, but you can read it too.  Like many of you, I have a near useless weather application on my phone. It tells me one of two things.  First, what is the weather now ? If I’m deep in the bowels of a client’s building in Stockholm or Glasgow, yes, I might want to know the weather now because I don’t know the local climate and glancing at my phone and seeing there’s a blizzard raging is useful.  But my consulting work is usually from home, so I can glance out the window.  The second thing my weather application tells me is the likelihood of rain at 3AM tomorrow morning. Unless I’m planning on being out and about at that time, I don’t care. But my phone assumes I do and I might see something like this:    ...  3AM  4AM  5AM  6AM  7AM ...    ...  13%  13%  18%  22%  8% ...    But what I want to know is whether or not I should send my daughter to school with an umbrella. Thus, I want to know how likely it is that that it will at at all tomorrow. So from the snippet above, is it likely that it’s going to rain between 3 to 7AM? I have no idea.  So how do I calculate that? Well, I am assuming that those percentages are for what are called “independent” events. What’s an independent event? Well, let’s say I want to know the odds of my dying tomorrow. You might assume that I want there to be no chance that I’ll die tomorrow, but that’s not true! I very much want there to be a chance of dying tomorrow because the only way there’s no chance that I’ll die tomorrow is if I’m already dead!  Thus, the odds of my dying tomorrow are dependent on whether or not I’ve survived until tomorrow.  But statisticians are boring and usually don’t use such colorful examples. Instead, they’ll ask a question like “what are the odds of flipping a fair coin twice and getting heads both times?” You simply mulitply the chance of getting heads by the number of times you flip the coin. You can quickly determine that the odds of getting heads twice in a row is 25%. These are also independent events, but they’re not rain.  If we assume that the percentage chances listed for each of those hours don’t take into account whether or not it’s already rained, we have independent event. So, what if, for each of the 24 hours in a day, the odds of rain are 95%? We can safely assume that it will probaby rain at least once tomorrow.  But what if the odds are only 10% for each of the 24 hours? That’s harder to know.  As it turns out, statisticians know very well how to calculate this and they use something called a binomial distribution  , but it’s not useful here because ...  If $p$ is the chance of success of an event, then the chance that we have exactly $k$ successes over $n$ independent trials is:  $$P(X = k) = \\frac{n!}{k!(n-k)!} \\cdot p^k \\cdot (1-p)^{n-k}$$  where $X$ is the number of times the event happened.  But that doesn’t actually help because we don’t want to know the odds of it raining once and only once. We want to know if it happens at least once. And that, it turns out, is easy. The odds of an event happening at least once is 100% minus the odds that it never happened.  The odds of something never happening are the chance of it not happening each hour, multiplied by the other chances of it not happening.  If there’s a 10% chance of rain at 2PM, there’s a 90% chance of no rain. That’s pretty clear, right? But if there’s a 10% chance of rain at 2PM and a 15% chance of rain at 3PM, what are the odds of no rain between 2 and 3PM?  $$(1-.1)\\cdot(1-.15)$$  That works out to roughly 77% chance of it not raining. That means there’s a 23% chance it raining between those two hours, even though the highest chance of rain it 15% at 3PM! Thus, even low percentages quickly add up.  Assuming that every hour has a 10% chance of rain, the chance of it never raining is $(1-.1)^{24}$ (90% multiplied by itself 24 times), or roughly .08, or 8%. Thus, the chance of it raining at least once tomorrow would be 92%.  Even though each hour only has a 10% chance of rain, I still need to send my daughter to school with an umbrella.  ","title":"Will It Rain Tomorrow?","url":"\/blog\/will-it-rain-tomorrow.html"},{"body":"  People do the TL;DR wrong. The summary should be at the top of the wall o' text, not the bottom. How else can you know if you want to keep reading? Maybe that makes it a “Too Long; Don’t Wanna Read.”  TL;DWR: I’ve resigned from The Perl Foundation’s Board of Directors.  This has actually been a long time in coming. In fact, I wrote this a couple of months ago. I started out, many years ago, answering a call for volunteers for The Perl Foundation (TPF). I became a grant manager. Eventually I wound up being the chair of the Grant Committee and at one point, took over our 2005 Google Summer of Code project when every Perl project was failing and I brought it under control with all projects being approved by Google. I was proud of that, though most have never heard of this.  I held on to the role of Grant Committee Chair for a few years before moving to the Steering Committee. Eventually, I wound up on the Board.  Along the way, I moved from the US to the UK, got married, moved from London to Amsterdam, had a daughter, moved to Paris, became a consultant and moved around a bit more. Now I live in the south of France, near the Italian border. Life has been a strange journey and volunteering for TPF has been part of that journey for most of my working life. Walking away from this is not easy.  I was asked by the chairperson of the Board, at one point, if I wanted to be vice-chair. I declined. I was asked (privately) at one point if I wanted to be the chair. I would simply propose a way forward and campaign for the role. I might have succeeded. I might have been the Chairman of the Board of Directors of The Perl Foundation. A prestigious title, even if it doesn’t mean much in the grand scheme of things. But still, that was a bit too much. If you’ll pardon the arrogant comparison, I prefer being Mr. Spock to Captain Kirk. I spent too many years as a child practicing in front of a mirror to raise a single eyebrow to give that up now.  Currently, the Board has members spread over multiple time zones and, in an attempt to find a time that worked for all Board members, the monthly Board meeting was moved to Friday evening, my time. This is the beginning of the weekend for me. It’s family time. It’s important and I don’t want to give it up. I couldn’t attend Board meetings.  I discussed other ways I could still contribute and the situation was understood, but I was falling behind in my understanding of Board issues because so many of them are handled in that meeting. Over time I stopped reading the meeting minutes. And that’s when I realized I was burned out. Or burnt out. Or whatever. I just didn’t care.  I had considered stepping down at that point, but dithered. Walking away from two decades of my life? That’s hard. And I felt like I would also be letting down friends in TPF, many of whom I’ve socialized with in numerous countries. These are people I know better than my friends here in France. This is a painful decision to make.  So let’s talk about the elephant in the room. Or more accurately, the CAT in the room. The controversy over the recent Community Affairs Team (CAT) action, the community’s response to that, the Board’s response to the community response, the community’s subsequent response, and so on … pushed me over the edge.  I don’t want to do this any more.  If you don’t understand the following, consider yourself lucky. I’ve no desire to explain this mess.  To be completely honest, I disagreed strongly with the Board’s final decision, though I voted to abstain, rather than “no.” I voted “abstain” because changes were made and I felt I would seem petty if I kept insisting “my way or the highway.” So I chose “abstain” over “no.” Call me a coward. You won’t be the first.  Did the person in question deserve sanction? Yes. Does the Board have the right to exercise authority over events that it funds\/organizes. Yes. Should the Board have applied the sanction to the individual? No, but only because the rules were not clear at the time of the offending behavior.  So I think what the Board did was wrong. However, the overarching goal of the action was to send a strong message to the Perl community: in those spaces where TPF has authority, TPF will act to ensure a safe environment for all participants. In terms of sending a message, it’s possible that the Board’s action was effective, even if it was wrong. I honestly don’t know the answer, though there’s a very loud contingent of people who will assure you that they do.  I must say that while I disagree with the Board’s actions, I am also disappointed by the response from many people in the Perl community. Some have been repeating untrue rumors, assuming they were true. Others have claimed the CAT team and the Board were acting in bad faith. This makes me sad. Mistakes have been made, but I saw no bad faith. I saw a bunch of good people—people I’ve known for years and consider friends—having to make a decision in a no-win situation. In this case I disagreed with the CAT decision, but I understand why they came to a different conclusion: the Perl Community has ignored toxic behavior for far too long.  To be clear, the CAT decision was more than just what happened at the event or on Twitter. There was a long pattern of inappropriate behavior. Some comments that I personally know about include “I stopped working on open-source project X because of Individual #1” and “I refuse to go on IRC because of Individual #1.” Driving people away from the community is exactly the sort of behavior a standard of conduct is designed to prevent.  Also, it’s true that Individual #1 has done brilliant things for the community. Some of the the most popular, go-to Perl tools are due to Individual #1. I recognize that, but we cannot give someone a pass for that. It wouldn’t be fair.  But I want to make clear that the CAT incident isn’t the reason I’m stepping down. I’m stepping down because it’s time. Fortunately, aside from the CAT incident, the Board is healthier than ever and under Stuart Mackintosh, I think it’s been doing great work. It pains me that the CAT mess overshadows everything else.  I’ve been part of TPF for too long. I’ve read too many profanity-laced emails, some sent privately to me. Some directed at me. I’ve seen horrible toxicity and complaints that we’re “woke” because we don’t want to support that toxicity. Fun fact: calling me “woke” doesn’t insult me; I wear that badge with pride. To me it embodies ideals that I strive for, even if I sometimes (often?) fail to live up to them. I’m trying to be a better person, too.  I’ve also seen incredibly, wonderfully supportive people. The overwhelming majority of encounters I’ve had with people in the community have been positive, if not downright wonderful. And I mean amazingly wonderful. I’ve seen community members struggling and other members reach out with support by job offers, money, or sometimes just kind words given at the moment they’re needed.  I see others who are no longer with the community. They’ve gone on to other things, but still receive love from the community. Still others have been pushed out because they can no longer handle the toxicity, but they’re still loved, too. Others, sadly, have passed on. (It still hurts knowing I’ll never sit in another pub in Belgium, being humbled by Jeff Goff’s understated brilliance, may he rest in peace).  Most people I know in the community are people I’m proud to know. We’re like a family: we disagree and we sometimes fight, but you’re wonderful.  So it’s time for me to stop rewriting this for the umpteenth time, trying to find the right tone. If I’ve learned anything from two decades with TPF, it’s that I can never quite find the right tone. Some people will agree with what I say while others will not. But that’s OK. That’s a community.  I’ve asked the Board if I can join the Advisory Board. It’s a group of ex-Board members who can offer advice (surprise!), but don’t get to vote. I expect they’ll approve, but they might not. Either way, I’m OK with that and I’ll move on and spend time with my wife and daughter, not feeling guilty about missing another Board meeting. I’ll miss many awesome people who volunteer their time supporting a community I love, but it’s time for a change.  Peace.  Update : I should mention that the individual who I said deserved to be sanctioned, but shouldn’t have because the rules were not clear at the time, has accepted full responsibility for their behavior and specifically agreed to the sanctions. They admitted they were in the wrong. I absolutely should have included that bit and I have nothing but respect for their acceptance not only of the decision, but of the reasons for it.  Update 2 : I have deleted comments which named individuals in question. There was a reason I chose not to name anyone.  ","title":"Resigning From The Perl Foundation","url":"\/blog\/resigning-from-the-perl-foundation.html"},{"body":"  What the fuck is that title? It’s not SEO-fodder. It’s not clickbait. It is incomprehensible. Any editor worth their salt would soundly reject this piece on the basis of that title alone. It is not an effective title.  I didn’t write that title for you. I wrote it for me. It’s from some writing of mine from 2007 which, to this day, tickles me pink about a magical time of my life. It’s my favorite title of all time and because it was written for me, it is an effective title.   I am, at heart, a writer. Two published books, one unfinished novel, a finished screenplay, several unfinished screenplays, numerous pieces for various companies, multiple blogs. I write. Even when I spend over a week researching and writing a single article   with only a handful or readers, I’m idiotically proud of it. I write as much for my own edification as for my readers. I don’t lay claim to being a great writer, but here we are, with you, dear reader, still reading.  I also write for pleasure and that is often at odds with writing for teaching...at least if you want your work to be read. If you want the world to find your magnificent crab cake recipe online, you have a boring title like “A Great Crab Cake Recipe,” or “The Best Crab Cake Recipe.” It’s easy for search engines to understand them. A title like “Good Shit In Your Mouth” might make you giggle but probably won’t resonate with many readers.  And as you write for others, blissfully ignoring rules such as “don’t start paragraphs with conjunctions,” you also have to consider if your audience is knowledgeable about your topic. One chef has remarked that when he translates American cookbooks into French, they are invariably shorter because his French audience knows how to cook; it’s their culture. They don’t need step-by-step details on how to sauté something and it would have been a waste of time, not to mention offensive, to explain it.  Another question: will your audience be sympathetic, open, or hostile to your point of view? Far too many writers ignore this. If you write an editorial justifying your belief that a particular president was terrible, the tone of that article can shift dramatically depending on whether that editorial is on the Fox News or The Guardian web sites. It’s the difference between being effective in swaying people or being smugly correct. Do you want to be right or do you want to be effective? You can’t always be both.  Understanding your audience is particularly painful when writing online: you might know your regular readers, but if some of your writing gets widespread, you’ve lost control over your audience and you had damned better well hope that what you wrote can stand on its own.  But there’s another type of writing. A different audience. An audience you know intimately. Yourself. You write for yourself while watching the words erupt from the keyboard in a flurry of paragraphs while you joyfully mix metaphors à la Barbara Kingsolver. Your write for yourself while sneaking a break, luxuriating in the beauty of the words of the late Carlos Ruiz Zafón.   I wrote the above for an unknown audience. I wrote the below for me.  March 2007. I was living in London, though “living” might be a stretch. I was living in a hotel bar with a few dozen of my closest colleagues.  In 2006, I had moved from the US to Nottingham, a small, lovely town a couple of hours north of London. In 2007, the company I was working for announced they were shutting down the Nottingham offices and if we wanted to keep our jobs, we had to move to London. To make a long story very short, the company put all of us up in a hotel for three months—a hotel in Hayes, a suburb of London once described by George Orwell as “one of the most godforsaken places I have ever struck.”  Case in point: shortly after we were herded into this hotel in the middle of nowhere, someone was murdered in the pub across the street from the hotel. The pub was packed, but there were no witnesses. We dubbed it “the murder pub.” We avoided that pub and either we made the long trek into London via the godforsaken Hayes and Harlington train station or we sat in the hotel bar, night after night, drinking. Startled guests would walk into the common area of the hotel to find my colleagues wearing pajamas, watching “footy on the telly” with a lager in one hand and the remote in the other (to prevent people from changing the channel). Living in this hotel is how I came to write the following, entitled “Gustatorial Adumbration and the Sheep of the Universe”.   “Where you from, mate?”  “The U.S.”  “‘Ere on holiday?”  How he missed the first ‘h' and caught the second is one of those mysteries of English speech that, like the game of cricket, is something Americans are Not Meant to Understand.  “No, I live here in London.”  “What’re you doin' in the hotel, then?”  “I sort of live here. It’s a long story.”  And it is a long story. Or a short one. All depends on where you start, really. I’ve spent several weeks at the Heathrow Comfort Inn Hotel, but I’m basically living in a bar with two or three dozen of my closest coworkers. I can’t really count them. Sometimes they’re here. Sometimes they leave for the weekend to go home to Nottingham, Beeston, or wherever else they have a real home, traveling to London only to put in their time in a soulless business park. And drink. Lots.  Earlier I was listening to a woman’s conversation in what passes for a restaurant at the Heathrow Comfort Inn Hotel. At £14.99 a plate, I would think they’d be able to hire a competent chef. Instead, they serve tasteless slop, devoid of spices, though if you’re lucky, you’ll get a tough, chewy papadam with it. A papadam is crisp; it snaps apart. Not these. More than once I’ve watched people struggle to tear one to pieces before shoving a bit in their mouth, their eyes widening in gustatorial   horror. It’s symbolic of the entire menu.  Curiously, my browser says the word “papadam” is misspelled. In searching through their spelling suggestions, starting, perhaps not inappropriately, with the word “escapades,” I see the word “adumbrate.” What the hell is that? Looking it up, I see that it means, amongst other things, “to foreshadow.” What a fucking perfect word. I don’t have a spell checker, I have a coauthor. I will never look at Firefox the same way again (and perhaps will be more careful about what I type to whom).  Getting back to the woman, she was nattering on about this and about that and her voice had a quality like an itch under the skin, one where you quietly sit there and scratch and scratch and get no relief at all. That’s because she had an American accent. Finding American accents irritating is not something I expected upon moving over here, but then I remember I have an American accent, too.  Sitting in my bar a few days ago, I was reading a book, turning page after page, desperately trying to figure out how our hero was going to survive my next whisky, when I realized I couldn’t figure out what language the ladies at the table to my left were speaking. Well, one of the ladies, actually. Two of them spoke English, but the third was chattering away on her cell phone and, jarred away from my book, I wondered what country she was from.    I wondered what her childhood was like. I wondered how she, too, came to be in this overpriced hotel—whose incredibly friendly staff is its only saving grace—when the awful truth hit me: she was Scottish. My brain switched to the “Scottish” filter and I could slowly make out what she was saying. She was talking about picking up some food from the store, but with a thick, burly accent that would as soon stick a knife in your kidney as say, “‘ere on holiday?”  I usually don’t have too much difficulty understanding the thicker accents. A couple of weeks ago, enjoying a night out at the downstairs bar with an Irishman, a Glaswegian (someone from Glasgow, Scotland), and an English lady. We found ourselves well in our cups, teasing the Englishwoman because I, the American, had no problem following the conversation but she kept asking them to repeat themselves because their accents were too thick.  Later the conversation turned, as it naturally does, to quantum physics and cosmology. David, the Irishman, was trying to explain to the English lady something about the “shape of the universe”, but we couldn’t stop laughing because his Irish accent kept forcing him to say “the sheep of the universe”. Eventually, superstring theory became superwool theory and we decided the universe is filled with billions and billions of subatomic sheep. God is my shepherd. The Bible, the definitive text on biology, will, in a pinch, fill in as a physics textbook.  Last week, about a dozen of us went to see the movie 300 at the IMAX theater in Waterloo. If you’ve ever been to an IMAX theater, you know these are the largest movie screens in the world and they have to have special film for their projectors. These things are huge. Which is why you might understand why I say that the naked women were not exactly titillating. Nipples the size of my head are not erotic.  I might add that there’s not much point in seeing this movie. It’s gay porn without gay sex. And like most modern porn, it’s all action, no plot. After reading about the Battle of Thermopylae, upon which the graphic novel was based (which in turn inspired the movie), I was surprised to discover the movie wasn’t entirely historically inaccurate. That, or the screenwriter consulted Wikipedia before setting fingers to keyboard.  The movie, in a nutshell, is 300 very buff men wearing little more than red capes and leather codpieces the size of a small car, spraying gallons of blood everywhere while somehow remaining mysteriously free of it themselves, even their blades. Oh, and a little bit of sex in the beginning just to make sure that guys have everything they want out of a movie. I am dumber for having seen it.  On the way home, Little Miss Drunk Off Her Ass stumbled onto the Tube and, after being told that she was about to sit on something wet and unidentifiable in one of the seats, asked if she could sit down in the middle of all of us. Though there were still plenty of extra seats on the Tube and inviting yourself to join strangers is simply not done , we said that was fine and proceeded to have very strange conversation with her. She was older and might have looked posh were it not for the blood vessels on her face making it clear how she spent her free time. After a bit of conversation about the movie and the buff men, she looked around at all of us and said, with a habitual drunkard’s precise enunciation, “I do prefer my men a bit thinner.”  Which is why I don’t really feel bad for what I said next.  “So, what do you do besides drink?”  The look of shock on her face was magnificent. Those were glorious times.  ","title":"Gustatorial Adumbration and the Sheep of the Universe","url":"\/blog\/gustatorial-adumbration-and-the-sheep-of-the-universe.html"},{"body":"    Introduction  Caveats  Life (as we know it)  Aliens?  Organic Compounds  The Drake Equation  What Is Life?  Life as Biopresentation  Philosophical Objections to Biopresentation  Why is rethinking life important?  Searching for Life Off-World  Unlikely Candidates  Triton  Ceres  Europa  Ganymede  Interesting Candidates  Enceladus  Venus  Mars  Conclusion     Introduction  The question of whether or not we’re alone in the universe is a profound one. There is something terribly sad at the idea that our star is the only one that shines down on life. With a possible lower bound of $10^{22}$ stars in our universe    and a universe almost 14 billion years old, that’s an average of almost two million stars created every day . While estimates of the number of planets vary, we think there are probably at least as many “Earth-like” planets as there are stars, that seems like two million chances per day of life being created. Of course, it’s much more than that because for every day that a planet doesn’t have life evolve, it has plenty of additional days, years, eons for life to evolve.  Given the preceding, many people are convinced that life is plentiful in the universe, but so far, we’ve never detected extraterrestrial life amongst the stars. Are we alone? Maybe. However, we should look closer to home. In fact, there are many intriguing clues in our own solar system which are worth examining. I’ll cover some of them, but I’ll save the best for last because it’s very exciting, even if we did overlook it for decades.  First, I want a to give a some background information for those who are curious about the search for extraterrestrial life but haven’t dug into the topic, just how complex it is. If that’s less interesting to you, you can skip ahead to the best candidates for life in our solar system .  Caveats  As with any writing on this topic, everything here is going to be a gross oversimplification. Unlike many other writings on this topic, I’m going to try to give a broad (but not deep) background into many of the issues involved. Sometimes I’ll provide additional caveats that might distract from the text, so you can click on the \"   \" symbol for more context.  Life (as we know it)  First, when I write “life,” mentally add the words “as we know it” after that. I get tired of typing it. The definition of life is muddled, to put it kindly, and we only have one tree of life as a sample. We have no idea what alien lifeforms might be like.  Aliens?    Not gonna happen. Source    Always remember that if the question is aliens, the answer is “no.” This might seem surprising to some who’ve attended my talks and heard me warn about those who use words like “always” and “never” in describing complex topics. People who use those words are often zealots. I don’t think I’m a zealot, but the default position of those who understand science is that extraordinary claims require extraordinary evidence. The existence of extraterrestrial life would be one of the most extraordinary claims of all time. So far, there’s no evidence of aliens. Thus, your default position should be “no aliens.”    Organic Compounds  You often hear the term “organic compounds” or “organic molecules” in connection with the search for life. Be careful.  In chemistry, organic compounds are just chemical compounds that contain carbon-hydrogen covalent bonds. In the popular press, we tend to use the term to mean any compound containing carbon, but it’s widely misinterpreted as “molecules related to life.” They are often related to life, but “organic,” in this context, does not mean life.  The Drake Equation  Before we can search for life, we have to define it. But before we get there, let’s think about the Drake Equation  . This was created in the 1960s by the late Dr. Frank Drake as a thought experiment . Like “organic compounds,” this is also widely misunderstood. I’ve often seen it referred to as an equation that helps us understand the prevalence of life in the universe. In reality, it’s a tool to help us understand the possible number of currently communicating extraterrestrial civilizations that SETI  might be able to detect. For the equation, the number $N$ is the number of civilizations we might be able to detect. The equation looks like this:  $$N = R_{*} ~ \\times ~ f_{p} ~ \\times ~ n_{e} ~ \\times ~ f_{l} ~ \\times ~ f_{i} ~ \\times ~ f_{c} ~ \\times ~ L$$  Those terms are:   $R_∗$: the average rate of star formation in our galaxy  $f_p$: the fraction of those stars that have planets  $n_e$: the average number of planets that can potentially support life per star that has planets  $f_l$: the fraction of planets that could support life that actually develop life at some point  $f_i$: the fraction of planets with life that actually go on to develop intelligent life (civilizations)  $f_c$: the fraction of civilizations that develop a technology that releases detectable signs of their existence into space  $L$: the length of time for which such civilizations release detectable signals into space   Depending on how you fill in those numbers, we could be alone in the universe, or we could have many advanced extraterrestrial civilizations within our galaxy.  For our purposes, we’re most interested in $f_l$: how often does life evolve? In fact, we’re not even interested in just planets because life could possibly evolve on a moon (there are several interesting candidates). Some panspermia  advocates even claim that life could have evolved in space, though this isn’t taken seriously by most scientists.  To be more specific, we’re asking “what are the odds of life spontaneously originating from simple organic compounds?”   This is known as abiogenesis  and is the subject of much research. The current consensus on an answer is, “we have no fucking idea.” What’s particularly maddening about this is that we’re not likely to have a clue until we detect other life. If we don’t detect life, that doesn’t mean it’s not plentiful (absence of evidence is not evidence of absence).  But if detect a biosignature  or technosignature  of life on a star on the other side of our galaxy, that might only put a lower bound on the amount of life out there, not an upper. We’ll need multiple detections of life to get a good sense of its ubiquity. Nonetheless, detecting any other life outside our solar system suggests that it’s common throughout the universe.  So now it’s time to ask the hard question: What is life?  What Is Life?  Again, this must be prefaced with the “life as we know it” caveat. And I’ll come right out and say what’s on my mind: most definitions of life are a steaming pile of shit.  One definition of life  , as taken from Merriam-Webster, is “an organismic state characterized by capacity for metabolism, growth, reaction to stimuli, and reproduction.”  Right off the bat we see a problem. Mules, for example, don’t reproduce.   Would you say they’re not alive? Of course not.  And what about fire? It reacts to stimuli and grows  . We could even argue that it metabolizes what it consumes for energy to keep growing, but we don’t think of it as “alive.”  There are other properties of life, however. Homeostasis  is the body’s ability to internally regulate itself and keep functioning. Does fire exhibit homeostasis? It’s hard to argue that it does. However, warm-blooded and cold-blooded animals have completely different approaches to homeostasis and homeostasis evolved over time  , so there are different levels of homeostasis. Maybe we can argue that humans have a homeostasis level of ten while fire has a homeostasis level of one? A piece of flint, however, might have a homeostasis level of zero.  But why would we argue that?  Life as Biopresentation  When I was working on ETL  systems to reduce the cost of Phase III clinical trials  , I often found myself working with data supplied by many pharmaceutical companies. Company A might refer to “Dr. Robert Smith, St. Mary’s Hospital, Arlington, Virginia.” Company B might refer to “Dr. Bob Smith, St. May’s Hospital, Arlington, Virginia.” Are they the same person? “St. May’s Hospital” might be a typo. “Bob” is short for “Robert.” Maybe they are the same person.  In reality, it’s hard to definitively state “yes” or “no.” Instead, I had access to tons of information about each researcher, including email addresses, phone numbers, ages, and so on. I would assign scores (how reliable) and weights (how important) to each piece of data and if two researchers crosses a certain threshold, they were probably the same researcher.  This same approach should be taken to the definition of life. Instead of life being a binary “yes or no,” we should describe its “biopresentation,”   or “how present are the signs of life in a ‘thing.'” So for a virus:         Property  Score  Weight  Value      Metabolism  1  5  3    Homeostasis  0  4  0    Growth  0  2  0    Reproduction  3  4  12    Communication  1  5  5     In the above, a virus might have a biopresentation of 20. A human, however, would be radically different:         Property  Score  Weight  Value      Metabolism  5  5  25    Homeostasis  5  4  20    Growth  5  2  10    Reproduction  5  4  20    Communication  5  5  25     A human might have a biopresentation score of 100.    This would be particularly useful in describing abiogenesis. One hypothesis of abiogenesis is  :   Prebiotic atmosphere with ammonia, carbon dioxide and water vapor.  Lightning produces simple organic compounds that fall into solution in shallow water.  The compounds react further in a prebiotic broth, forming amino acids.  The amino acids link up with peptide bonds to form polypeptide chain proteins.  The proteins combine into more complex molecules that can replicate and metabolize simple substances.  Complex molecules and organic compounds form lipid membranes around themselves and start acting like living cells.   At all steps along the way, we could have something closer to what we call “life,” though the initial steps might have a biopresentation level of zero. Instead of life being a fixed point, it’s a gradual process. We don’t point to a thing and say “this is alive” because that gets us back to the arguments of what life is or isn’t. We simply describe its biopresentation.  This would be useful for biology today, too. Consider the endless arguments about whether or not viruses are live  . It’s often argued that viruses aren’t alive because they can’t metabolize outside of their host. That reduces metabolism to the primary (or sole) attribute of life.   But metabolism is the global term for anabolism and catabolism  . Catobolism is just breaking complex stuff into simpler stuff while anabolism is using that simpler stuff to do more complex stuff. Fire catabolizes, but doesn’t anabolize. Viruses do both. So maybe they’re alive? But some still reject that because, again, they can’t do that without a suitable host.  But what’s a host? Obligate parasites such as the protozoan Plasmodium   can’t survive outside of their hosts. Or consider humans. If I drop you into the middle of interstellar space without a space suit, you’ll stop metabolizing, too. So the context in which we can biopresent is also a spectrum. Viruses are starting to look more alive.  But wait! There’s more!  Most living beings have some ability to communicate with one another. Maybe all do. As it turns out, we’ve know for a few years that viruses can do this, too  . The human communication score might be at a “ten,” while viral communication might be a “one,” but fire’s communication is “zero.” The weight of this factor is probably pretty high, so fire’s biopresentation would be pretty low, but viruses could be easily crossing the threshold.  Pragmatist that I am, I want to skip the philosophical debates about whether or not something is alive and go with the pragmatic discussion of how closely something resembles life. However, we can’t ignore the philosophical implications.  Philosophical Objections to Biopresentation  At this point, I can see many biologists howling at the above. In a 2013 Wired article entitled It’s (Almost) Alive! Scientists Create a Near-Living Crystal  , the lede reads:   Three billion years after inanimate chemistry first became animate life, a newly synthesized laboratory compound is behaving in uncannily lifelike ways. The particles aren’t truly alive — but they’re not far off, either.   The crystals created had metabolism and mobility, but couldn’t reproduce. You can read the original study at The Royal Society website  .  For these crystals, one could argue that a), they’re not alive, or b), what their relative biopresentation level is. Some people would be uncomfortable describing crystals as alive, but if we’re to search for life elsewhere in the universe, are we going to argue that life which doesn’t share the same biochemical basis as our own isn’t alive?  Or consider self-replicating Von Neumann machine  . By the biopresentation argument, they would likely score very high, but are machines “alive”? Or apply biopresentation to The Gaia Hypothesis  . Is our planet alive? That’s definitely a philosophical question, but in light of biopresentation, depending on what factors to include, we could make a strong argument that it is.  Why is rethinking life important?  What does this have to do with searching for life off Earth? It’s simple. If we go with a binary “yes\/no” for life, it means that we have defined a limited, possibly incorrect, set of properties necessary to define life. Something not meeting those conditions might be disregarded. If, however, biopresentation is adopted, we can start defining “lifelike” properties and perhaps not be so quick to dismiss something that is otherwise alive.  If that seems stretched, consider the “Shadow Biosphere,” an idea that there may be another biosphere on Earth that we’ve not yet detected. The idea’s not popular, but it seems persistent. Here’s Fraser Cain, the founder and publisher of Universe Today, describing it.     Ultimately, if there is a shadow biosphere on Earth and we can’t detect it, this doesn’t bode well for detecting life off Earth.  Searching for Life Off-World  We can search for biosignatures in other star systems, but until we can visit   , it’s likely to be controversial. Technosignatures are more promising because a receiving a radio transmission of a sequence of prime numbers followed by an encoded message is pretty conclusive. But in the absence of technosignatures, we need to fall back on biosignatures and in our own solar system, we can definitely visit.  First, let’s remove a few unlikely candidates. They’re candidates because we can envision life being there. They’re unlikely because it may not be “life as we know it” and thus are less likely to recognize it as life. We might even exclude them for a practical reason: it’s hard to visit them with current technology and we’re unlikely to be able to detect life if we do so due to the limited equipment we can bring.  Unlikely Candidates  If you’re looking for information on extraterrestrial candidates for life in our solar system, they’re plentiful. I’m of the opinion that we should investigate all of them, given how hugely important I value the idea of finding a second (and third, and fourth, ...) abiogenesis, but as a practical matter, we need to consider the cost (biosignatures on Pluto wouldn’t necessarily justify the resources needed to launch a mission there) and whether we have detected biosignatures. When it comes to searching for life, low-cost targets with biosignatures must be our first priority.  Triton    Neptune’s moon, Triton Source    The moon Triton orbits the ice giant  Neptune. It’s the largest Neptunian moon and along with the Jovian moons Io and Europa, along with the Saturnian moons Titan and Enceladus, is one of the five geologically active moons in the solar system  .   Triton is fascinating because it orbits Neptune opposite of Neptune’s orbit (it’s the largest moon in the solar system to do so) and it might have an ocean underneath its ice covering. This suggests that ocean existed prior to it being captured  , which might make that ocean older than Earth. However, it takes years to get to Saturn  and the amount of fuel necessary (and the complexity of slowing down once you reach the planet) limit out ability to look for life. Given that the extreme cold suggests that life can’t exist, it’s probably a low priority for us.  Ceres  Ceres, orbiting in our asteroid belt, is the largest dwarf planet. It might have an underground saltwater ocean  . It might also have abundant organic compounds  . However, many things unrelated to life are “organic compounds,” so this isn’t particularly inspiring. If the ocean does exist, we don’t know how it came about and the data is still unconfirmed. Ceres is interesting and accessible, but as a candidate for life, it seems poor.    Europa  Yeah, I’ve seen the dive Europa  art. This moon of Jupiter is tiny, but it apparently has more water than all of Earth’s ocean’s combined. It almost certainly has an ocean beneath its ice crust  , but I rate it a poor candidate in the search for life because its constantly getting blasted by radiation from Jupiter and we’ve seen no biosignatures. The ingredients for life seem to be there, but I’d rather we focus our attention on worlds which are giving us hints something is there.  Of course, 2010: Odyssey Two  , by Arthur C. Clarke contained the warning “ALL THESE WORLDS ARE YOURS – EXCEPT EUROPA. ATTEMPT NO LANDING THERE.” I’m just the sort of guy who would be tempted to visit just because I’m told not to.  Ganymede  The Jovian moon Ganymede, the largest moon in our solar system, is particularly intriguing because it’s large, it has a subsurface ocean, it has a magnetic field to ward off Jovian radiation, and appears to have many of what we think are likely prerequisites for life. This article by astrophysicist Ethan Siegel lays out a case for investing Ganymede for life  . However, I’m unconvinced for the same reasons as Europa: no biosignatures have yet been detected.  Interesting Candidates  Enceladus    Saturn’s Moon, Enceladus Source    At first glance, Enceladus does not seem like a likely candidate for life as it’s a small, icy moon of Saturn. However, it’s geologically active, with cryovolcanoes (they spew water), a subsurface ocean, and probably hot vents in those oceans. More importantly, something is pumping out a lot of methane from that ocean  . Methane is considered a biosignature  because it’s usually generated by life (on Earth, of course) and tends to degrade quickly, meaning that a large amount of methane suggests that something is replenishing it.  The team that analyzed the methane data studied a geological process known as serpentinization  which could generate methane, but came to the conclusion that it would be impossible to generate as much methane as was seen. A sample return mission was proposed for Enceladus  , but sadly, NASA did not accept the proposal. Enceladus is a wonderful candidate for sample return because ...   Its low gravity makes the mission easier  Methane is a biosignature  The cryovolcanoes spew the samples high enough to easily collect on a flyby  The potential for discovering life makes this a very high reward mission   We’ll probably have to wait a few decades for the answer to this one, which is why I was tempted to put it in the “Unlikely Candidate” category (e.g., pragmatism).  Venus  Venus, despite having a hellish surface, has long been viewed as a candidate for extraterrestrial life. In 1967, Carl Sagan argued that life was unlikely on the surface of Venus  . However, in the same year, he argued that life might exist in the clouds of Venus  . We’ve also found dark streaks in the Venusian atmosphere that can’t be easily explained  . We have no idea what the dark streaks are. We don’t know their origin or composition. We don’t know why they don’t mix with the rest of the atmosphere and we don’t know why they absorb ultraviolet light. From the article:   However, in a region beginning around 31 miles (50 km) in altitude and extending 7.5 miles (12 km) outward is a sweet spot where the temperature ranges between 86 degrees F and 158 degrees F (30 degrees C and 70 degrees C), and the pressure is similar to that on Earth’s surface. Life could potentially survive in this zone where the dark-streaking UV absorber is found.   Microbial life could easily explain those dark streaks. Further, we now know that the unusual Venusian magnetic field is created by an interaction between the solar wind and Venus' ionosphere  . Venus' ionosphere starts about 120 kilometers above the ground, so we have a magnetic field to help protect potential life against cosmic rays. Without such a field, those rays could sterilize the planet.  But again, aliens? No. There’s probably a non-biological explanation (there always has been, once it’s found). However, until life is ruled out, Venus is relatively easy to visit (easier than Mars, actually), and if a sample return mission could be arranged, it’s worth the effort. Even if we didn’t find life, the scientific knowledge we would gain would be tremendous.  More exciting is the recent detection of phosphine in the atmosphere. Since phosphine is generally produced by life on Earth, and we struggle to describe any geological process on Venus that could produce phosphine, the news created a lot of excitement because phosphine is a considered to be an excellent candidate for a biosignature in anoxic (low oxygen) atmospheres  . In particular:   Phosphine is a promising biosignature gas, as it has no known abiotic false positives on terrestrial planets from any source that could generate the high fluxes required for detection.   Phosphine on Venus would be stunning news, but others have come out to say there’s no phosphine after all  , and others have disputed the original research  led by Professor of Astronomy Jean Greaves  and her team at Cardiff University in Wales. She’s had to post follow-up papers to explain flaws in some of the objections to her work  . The overall consensus in the popular press seems to be that there no phosphine detected in the Venus atmosphere and Greaves and her team simply misinterpreted the data.  If Greaves is right, this could be huge news. If she’s wrong, that’s how science works. However, in what should be a major story, but has largely been unreported in the press, The Planetary Society  has published a recent podcast with Professor Greaves  where she announces that another telescope has apparently confirmed the existence of phosphine  , meaning two telescopes and at least three different instruments confirming its existence. This is not proof (and we need more teams than just Greave’s team reporting this), but it certainly suggests we need to follow this up.  For more information, check out the Life on Venus  page on Wikipedia.  Mars    The planet Mars.   If I could only choose one place in the solar system to search for extraterrestrial life, it would be Mars, hands down. However, my reasoning is due to some research which, sadly, has been largely ignored. It was research conducted in the 1970s, but we’ll get back to that.  We now know that Mars was relatively like Earth early in its history, with a large ocean covering its northern hemisphere and a generally warm climate. However, due to the lack of a magnetosphere, Mars' atmosphere was stripped away by the solar wind  . With that, it lost the ability to trap heat and it turned into a cold, apparently lifeless rock. We don’t know the exact timing, but it may have been about 1 to 1.5 billion years ago that Mars lost its magnetosphere  , giving life plenty of time to evolve, given that life was well-established on Earth at this time.  However, that’s not all. Methane has possibly been detected on Mars, but the Trace Gas Orbiter did not find any methane  . However, some recent studies suggest that there may still be methane produced on Mars  by unknown sources. The existence of methane on Mars is still highly controversial, but I consider three things:   Mars was very Earth-like for billions of years  We might have detected a key biosignature  We are considering colonizing Mars   Those three things together make Mars a strong candidate for searching for life simply because we can do this as a side-effect of other ongoing efforts.  That is not, however, the primary reason we should choose Mars as our primary candidate in the search for life. Yes, we have a potential biosignature, but you might remember my biopresentation discussion above. We might have discovered metabolism on Mars, and that happened back in the 1970s.  From this phys.org article  :   In 1976, two Viking landers became the first US spacecraft from Earth to touch down on Mars. They took the first high-resolution images of the planet, surveyed the planet’s geographical features, and analyzed the geological composition of the atmosphere and surface. Perhaps most intriguingly, they also performed experiments that searched for signs of microbial life in Martian soil.   In the experiment, these two landers, thousands of kilometers apart, each injected a $C_{14}$-laced (carbon 14) nutrient solution and then examined the air. They each detected that something had released the $C_{14}$ from the solution. They also heated each sample of 50 degrees Celsius (122 degrees Fahrenheit) and repeated the experiment. Each detected significantly reduced amounts of $C_{14}$, as would be expected if the heat killed off organisms.  Previously, this experiment had been repeated extensively on Earth to verify that it was indeed a solid indication of life. It was also tested with known sterile samples to see if some other effect were causing the release of $C_{14}$.  So in 1976, we had a strong indicator of life, or “biopresenter,” as I like to think of it. However, there were some problems. The biggest obstacle could be if some natural chemical process could replicate these results. A type of salt called “formate” could possibly be responsible, but it’s not thought that formate is heavily present on Mars. Perchlorate, which we know is heavily present on Mars, can produce similar results, but it does so even when heated to above 160 degrees Celsius, so it’s not consistent with what we’ve seen. Hypochlorite might be present on Mars and it breaks down when heated to above 160 degrees, but its behavior has not been tested at 50 degrees,   . It’s currently the best explanation for the observed biopresentation, but even if the 50 degree test is carried out, that doesn’t rule out microbes on Mars. Of course, I remind you of the answer to the “aliens?” question. Read the article in full to understand more  . It’s a complex question and there are no clear answers.  Conclusion  For Mars, we had suitable conditions that life could have evolved in. Given the ubiquity of extremophiles on Earth, it’s not a stretch to imagine ancient Martian life evolving to exist in their current harsh climate. We have a potential biosignature and a potential biopresenter. Mars is our best bet for looking for extraterrestrial life.  However, there are so many other interesting questions here. Recent research bolsters the case for panspermia  , so if we discover life on Mars, did it originate on Earth? Or, since we know Martian meteorites have landed on Earth, are we Martians? I certainly hope not. I have nothing against the idea of of originating on Mars, but I want to find many examples of abiogenesis. That would tell us that our universe is probably filled with life. The more life there is, the more chances for intelligent life to exist somewhere else. Of course, that brings up the Fermi Paradox.  With the the new space industry taking off , it’s a very exciting time to be alive and exobiologists might turn from biologists who write science fiction to one of our most important scientific disciplines.    It’s Time.   ","title":"Searching for Extraterrestrial Life in Our Solar System","url":"\/blog\/searching-for-extraterrestrial-life-in-our-solar-system.html"},{"body":"  This blog entry is actually for my daughter. She’s eight years old and already turning into a math wizard. While other children are struggling with simple multiplication, she’s delving into square roots, negative numbers, factorials, and so on. She’s also been asking her teacher at school when they’re going to start getting problems in base 5 (she can count in different bases &lt; 10 now).  So when we were discussing factorials, she asked why the factorial of zero is one and I, um, stumbled. I was a math whiz in my high school, taking calculus in my jumior year, surrounded by seniors, but I’ve forgotten most of my math and for the life of me, I couldn’t remember the answer to her question. So here it is, after a bit of a refresher.  You probably remember that the factor of an integer, n , is the product of all integers from 1 to n . So we would say that that the factorial of 4, or 4!, is equal to:  $$4! = 4\\times3\\times2\\times1 = 24$$  And for a few other numbers:  $$3! = 3\\times2\\times1 = 6$$ $$2! = 2\\times1 = 2$$ $$1! = 1 = 1$$  But where does that leave the factorial of zero? There doesn’t appear to be any room left to put that equation.  There are two simple ways of answering this. One is intuitive and the other involves simple math.  The Intuitive Solution  Imagine that you have three runners in a race, Alice, Bob, and Charlie. Assuming ties are not allowed, how many ways can you arrange them into first, second, and third place? Mathematically, we’d ask “given a set of n elements, how many different ways can you order them?” That’s what’s called a “permutation.”  Another way of describing a permutation is “how many ways can you rearrange a set of items?”  Well, let’s work it out the hard way:   Alice, Bob, Charlie  Alice, Charlie, Bob  Bob, Alice, Charlie  Bob, Charlie, Alice  Charlie, Alice, Bob  Charlie, Bob, Alice   So for three participants, there are six different ways we can order them. Not coincidentally, that’s equal to 3! (three factorial). What if it’s only Alice and Bob?   Alice, Bob  Bob, Alice   That’s 2! (two factorial).  If you are really bored, you can work out four participants, but I promise you that it’s equal to 4! (four factorial).  So to figure out the number of ways you can order n elements, you just calculate n ! ( n factorial).  But what is n ? In math, we have the concept of a set .   A set is merely an collection of n distinct things. The order of those things doesn’t matter. The set {Alice, Bob, Charlie} is the same as the set {Charlie, Alice, Bob} . So our original question is “how many permutions of a set are there when n = 3?” And the answer is 3! (three factorial).  But it turns out that an important concept in math is the “empty set”, designated as {} . How many permutations does it have? Or to ask another way, “how many ways can we arrange nothing?” 1 and only 1. Thus, 0! = 1.  Do the Math  OK, that sounds like cheating, so let’s look at it another way.  As it turns out, we can say the following:  $$n! = \\frac{(n + 1)!}{n + 1}$$  How does that work? Well, let’s use 4! as an example.  $$4! = \\frac{(4 + 1)!}{4 + 1}$$ $$4! = \\frac{5!}{5}$$ $$4! = \\frac{5\\times4\\times3\\times2\\times1}{5}$$  And since the fives cancel out:  $$4! = 4\\times3\\times2\\times1$$  And working this out with other numbers:  $$3! = \\frac{(3 + 1)!}{3 + 1}$$ $$2! = \\frac{(2 + 1)!}{2 + 1}$$ $$1! = \\frac{(1 + 1)!}{1 + 1}$$ $$0! = \\frac{(0 + 1)!}{0 + 1}$$  Since it’s 0! that we’re interested in, we see it works out to:  $$0! = \\frac{1!}{1}$$  Since 1! is 1, we have:  $$0! = \\frac{1}{1}$$  And one over one is one. Thus, 0! = 1.  ","title":"Why is zero factorial equal to one?","url":"\/blog\/why-is-zero-factorial-equal-to-one.html"},{"body":"    Introduction  The Tyranny of the Rocket Equation  How to Get to Mars  How to Get to Mars, Extra Credit  The Physics of Perseverance  Rocket Engineering  Rocket Fuels  Nuclear Thermal Rockets  Max q and Bernoulli’s Principle  Re-entry and Hypersonic Flight  Conclusion     Introduction  The old space race was driven by governments, desperate to bolster national pride. The new space race, known in the industry as “NewSpace”, is driven by entrepreneurs, looking to make record profits. Science fiction is replete with stories of the horrors of corporate space colonies, charging workers for the very air they breathe and denying them basic rights. These seem far-fetched, but our history shows the dangers  . Even if well-intentioned, what happens when you’re living in a mining colony asteroid owned by Asteroids, Inc. and they go bankrupt?  There’s plenty to be concerned about, but nonetheless, we’re charging into this future and I confess, I love it. My sister, Gayle, taught me to read at a young age and while in the first grade, other students would be sounding out “see Spot run” while I was reading National Geographic. At home, my mother had stacks of old pulp magazines such as Astounding Magazine, The Magazine of Fantasy and Science Fiction, and so on. Ever since I can remember, I’ve dreamed of the stars and, despite the concerns, I’m delighted we’re finally taking the next steps to get there.  But how does it all work? Why is it so hard? What the hell is “max q”?  The Mars Society of Canada  has published a series of blog posts by Jin Sing Sia  about rocket physics. They’re introductory articles, but they’ll get you up to speed pretty quickly on the basics. You don’t even need to know advanced math.  There is, however, a slight problem with the articles. They didn’t include a table of contents! We don’t even have “previous” or “next” links for the articles. So allow me to do that, along with providing a summary for each. Note that these articles should probably be read in order, but it’s not strictly necessary.  The Tyranny of the Rocket Equation  link   If you follow space news, you may have heard about the tyranny of the rocket equation. In short, if you have a small amount of fuel, your thrust won’t last long enough for you to leave the planet. However, if you add more fuel, it’s not just lifting the rocket, it’s lifting the rest of the fuel! The more fuel you add, the harder it is to lift the rocket.  How to Get to Mars  link   Ever wonder why Mars only has “launch windows” every 26 months? What’s a “Hohmann transfer orbit”? Can we get to Mars faster than the projected nine months?  How to Get to Mars, Extra Credit  link   Discover the basics of orbital mechanics. They were touched on in the previous article, but this one goes a bit more in-depth. You can skip it if you want, but why would you want to?  Light physics and mathematics knowledge is recommended. Maybe you’ll finally learn what “perigee” and “apogee” are!  The Physics of Perseverance  link   By now, you know the Perseverance rover is cruising around Mars, taking selfies, doing amazing science and generally having a blast—up until the point when she realizes she doesn’t have a ride home.    The Perserverence Rover on Mars took a selfie.   The fact that Perseverance was able to land on Mars at all is amazing. The “sky crane” that lowered her to the surface was a thing of beauty. This article explains the approach, atmospheric entry, descent, and landing.  Rocket Engineering  link   Did you know that your car’s engine is more mechanically complicated than most rocket engines? On the other hand, your car’s engine doesn’t burn hot enough to melt the engine. Learn why rocket science ain’t easy.  Rocket Fuels  link   Liquid hydrogen is one of the best fuels for rockets, so why don’t more rockets use it? What is hydrazine? Why do we use hypergolic fuels when they’re so dangerous? Wait, what’s a hypergolic fuel?  Nuclear Thermal Rockets  link   Nuclear rocket engines are amazingly powerful. Their exhaust also tends to be amazingly radioactive. And if a launch goes poorly, you can’t just use your FTS (Flight Termination System) to blow it into little pieces lest you shower everyone with radioactive waste. So why are we considering them?  Max q and Bernoulli’s Principle  link   Max q and Bernoulli’s Principle are two terms you often hear when space nuts discuss rockets. Understanding them helps to explain, amongst other things, why the opening scene to the otherwise excellent film (and book), “The Martian”, was so unrealistic.     Re-entry and Hypersonic Flight  link   You know space capsules have heat shields to protect them from the heat of reentry to our atmosphere, right? It’s not because of friction.  You’ll also learn why sound travels much faster under water.  Conclusion  Space is amazing. And it’s hard. I hope you enjoyed reading through all of the linked articles.  The NewSpace race is well under way and we’re finally going to get to the planets, even if we don’t colonize them. I am hopeful, however, that eventually we will.  ","title":"Rocket Physics 101","url":"\/blog\/rocket-physics-101.html"},{"body":"","title":"Why Black People Hate Your Dreads","url":"\/blog\/why-black-people-hare-your-dreads.html"},{"body":"  Defund The Police?  The phrase “Defund the Police” has flowed across the US like a tidal wave, wrecking all conversations in its path. In the wake of the murder of George Floyd , public anger has reached a point where it’s tearing the US apart. While it would be too simplistic to claim that police brutality is primary driver beyond the discontent in America today, it cannot be ignored. However, “defund the police” has to be one of the most idiotic rallying cries people have rallied behind.    Police in riot gear. Source    This is largely because defunding the police isn’t what the protests are trying to accomplish  . Instead, the movement is about allowing police to be police, dealing with crime, and not having SWAT teams serving eviction notices  . So if a homeless guy urinates on a sidewalk, trained social workers can evaluate the person, determine if there’s an actual threat, and assist them in getting help, along with gaining access to appropriate services.  Or just send out men with guns who receive little training in social issues and who will toss the homeless person in jail. I’m not sure how that helps anyone. Urinating in public puts you at risk of a year in jail (two in Ohio) and being placed on the sex offenders registry  . Is this a proportionate response? Is destroying someone’s life a healthy response when the alternative is to offer to help the person get back on their feet?  But instead of a slogan such as “let police be police” (in my reading of police forums online, many police support this), many are calling for defunding the police, which those on the other side disagree with because they think it means “eliminate the police,” despite many people trying to clarify the meaning. As anyone in marketing can tell you, if you have to explain what you meant over and over again, then clearly what you said communicates something different from what you meant.  But what‘s the alternative? All but the most authoritarian amongst us admit that there is a problem; it’s the solution we disagree on. Is there a better way?  I’ve been reading through the blog of Donald DuPay  , a former Portland police officer. In particular, he wrote Bad Training and a Culture of Lying Continue to Haunt PPB  . While much of the language is strong (for example, the “n” word is sometimes used, but in a clear context), if his account is true, he lays out a strong case that there are serious issues with policing. In my reading of police forums, I can only attest to this. What DuPay does, however, is interesting. In particular, he cites the 14th Amendment of the US Constitution  , of which Section 1 reads, in part:   [No state shall] deny to any person within its jurisdiction the equal protection of the laws.   Internal Affairs  What is “equal protection”? As DuPay argues:   It does not state for instance, that electricians may have their own private justice system. It does not state that plumbers may have their own justice system. And it does not state that the police can have their own private justice system either, and Internal Affairs is a private justice system for the police, which make it unconstitutional.   DuPay cites many instances of Internal Affairs (IA) investigating criminal behavior of police officers and giving them a slap on the wrist. Or at the very least, giving them punishments far less severe than what the public would face, but still allowing the offending officers to keep their jobs. Officer DuPay cites instances of officers being investigated by IA and being allowed to keep their jobs, despite committing theft or drunk driving, and further claims that:   Over 80 percent of cases that go before IA are whitewashed away and swept under the proverbial rug, with officers either not being charged or given flimsy disciplinary actions for serious infractions involving police misconduct.   And what would the alternative be? Simple: let the police go through the same justice system anyone else has to go through. If they’re found not guilty, let their union or jurisdiction pay for their defense. If they’re found guilty, they pay for their own defense and they suffer the same penalties non-officers would face, with the possibility of greater penalties if they’re found to violate Section 242 of Title 18  which   makes it a crime for a person acting under color of any law to willfully deprive a person of a right or privilege protected by the Constitution or laws of the United States.   Of course, there are problems with this. Sometimes a police officer might engage in illegal activity as part of undercover work  . For example, a police officer might receive stolen goods and later fence them in order to break apart a criminal organization. The buying and selling of drugs is another example: if you can’t show the criminals have willingly engaged in the activity ( assuming you are not engaged in entrapment  ), then it can be very hard to secure a conviction.  Or maybe the case against the officer requires exposing a confidential informant whose life may then be jeopardized? Again, this one is difficult because when laws are being broken, questions of right and wrong can become murky. However, Human Rights Watch has researched IA divisions and found  :   Internal affairs divisions must be central to any examination of how police departments deal with abusive behavior by officers. Therefore, it is alarming that no outside review, including our own, has found the operations of internal affairs divisions satisfactory. In each city we examined, internal affairs units too often conducted substandard investigations, sustained few allegations of excessive force, and failed to identify and punish officers against whom repeated complaints had been filed. Rather, they, in practice, often shielded officers who committed human rights violations from exposure and guaranteed them immunity from disciplinary sanctions or criminal prosecution.   And honestly, this is probably a kindness to many IA officers. Typically they don’t spend their entire career in IA. Instead, they were once regular police officers and they’re now called on to investigate their friends and colleagues. The conflict of interest is clear and it must be excruciating. Just consider the well-known problem of steroid abuse by police  . They have a very dangerous job and if they are afraid to to subdue a violent suspect, they may do what many police officers are doing: use steroids and hit the gym. And you, as an IA officer, formerly a beat cop, may have tremendous sympathy for them. And you also know you won’t be an IA officer forever and will be working with other police officers again. For many, this can be an impossible situation.     What I find appealing about this solution is that the IA officers don’t lose their jobs: they simply revert to be regular police officers. Further, it leverages existing systems rather than requiring the creation of new ones. Additional safeguards will need to be in place to account for the unusual nature of police work, but why should an officer who passed out drunk, on duty, in their car, get to keep their job, much less escape the criminal sanction everyone else would face  ?  I “get” the motivation behind “Defund the Police.” More than that, I completely support it. But there comes a moment when we need to distinguish behind what we should do and what we can do. Maybe getting rid of Internal Affairs and having the police face the same justice system as everyone else is an effective compromise. It won’t fix the problems, but it could be a start.  ","title":"Don't Defund The Police","url":"\/blog\/dont-defund-the-police.html"},{"body":"  It’s a testament to my late father that he managed, even in death, to surprise me. While recently in Germany for Jim’s funeral, I discovered that Jim admitted to my sisters that yes, we probably have a sibling or two in Russia. Given Jim’s propensity for mendacity when discussing his children (in particular, denying their existence), his admission was a surprise.    Jim’s ashes.   Just to recap for those not following the story: our father, Jim, had—that we know of—seven children by four women. Aside from children who had the same mother, we didn’t know about each other and it was many years of work to find out about each other’s existence and eventually locate each other. We’ve grown very close, despite our father, and now it turns out there may be “one or two” more, probably born near Moscow in the early 70s. That’s not much to go on.  So what do we have to go on?  KGB  I’ve been told that the KGB would have had, at the very least, cursory files on any Americans living in the Soviet Union during the height of the cold war. Given Jim’s background and the fact that he spoke fluent Russian, he would clearly be a person of interest. The KGB files might have information about Soviets that Jim knew (“knew” being such an overloaded word there). Though I’ve a few Russian contacts, I assume that any such records are not available to me.  Janath Poe  While in the USSR, Jim was married to woman named “Janath Valentine Poe”, possible family name of “Heritage”, probably British. I haven’t found her. She might have more information. (I’ve found hints of her online, but I don’t know if it’s the right person and I don’t have contact info).  Dennis Blewitt, Daily Mail  One of Jim’s friends was Dennis Blewitt, a journalist working for the Daily Mail. Blewitt was later expelled from the USSR in 1985  in apparent retaliation for the UK expelling Soviets accused of espionage. I’ve reached out to the Daily Mail a few weeks ago, but have not heard back from them. I was hoping they could put me in touch with Mr. Blewitt and that he might have some recollection of who Jim knew.  The Letter  In Jim’s effects I found a letter from Sweden written to Jim in 1978. I shared that and some other information on Reddit and a sharp-eyed reader  managed to make out a nearly illegible signature and produced the names “William Aldridge” and “Belkacem Bazi”. These two aspiring filmmakers went on to make a number of films together. Given some clues in the letter, I found that the writer, Aldridge, likely knew Jim when he lived in the Soviet Union. Aldridge wrote:   In any case, I have no forgotten my debt to you, nor has Belkacem who sends you his regards.   Sadly, further research has revealed that both of these men have passed away. Short of me finding family members who know something, I’ll never learn exactly what assistance Jim offered to these two men. At the time of the letter, Aldridge was in Sweden and Belkasem was in Canada. There’s no suggestion of a connection with the Soviet Union, though in 1999, these two men collaborated on a French series about Russian cinema.     My sisters, Lynne and Gayle, sitting at the Idsteiner Brauhaus, in Idstein, Germany. It was Jim’s favorite watering hole.   So that’s where we’re at. Aside from Janath Poe and Dennis Blewitt, we have no leads. Jim, to the very end, provided no help. Going through his documents has also provided little help, but we’ll press on. We have more siblings to discover!  ","title":"Missing Siblings?","url":"\/blog\/missing-siblings.html"},{"body":"  The Stupid Experiment  I started a “Facebook experiment” after one of my posts went viral with millions of views. It was a lovely video of excited dogs at a shelter joyfully picking out their own Christmas toys.     With this temporary internet fame, I decided to expand my circle, to engage with those I would not ordinarily engage with, to respectfully counter opinion with facts (and references), and to try to understand those who have differing points of view. In short, I was deluged with friend requests and I accepted almost all of them.  I can now share the results of that experiment: “People on Facebook are terrible.”  There are exceptions, but generally, the pattern has been this:   Someone posts something easily disproved  I politely inform them that they may have encountered “fake news”  I provided links (trying to use sources they would view as credible) to give them more information   The result, in almost every case, is that I was ignored and they continued posting fake news.  In a few cases, abuse was hurled at me. For politely providing correct information, I was called an “idiot”, a “troll”, a “socialist”, and, one one memorable occasion, a “Nazi on the payroll of George Soros.” Given that Soros is a Jew who survived Nazi-occupied Hungary, the last was particularly sad.  And once, I received a polite “thank you” and once a post was taken down in response.  Very few people were willing to engage constructively with me. Only one provided references to back up their point of view. I thanked them for that.  In short, the experiment was a failure. Facebook is clearly a platform that people use to vent, not to learn. And frankly, the discussion was depressing. So I’ve stopped.  I’ve started to unfriend those who are particularly bad. Generally this was easy. Obvious bots and those posting “Obama is a Muslim” memes are easy to discard.  But one was an old friend who kept posting blatantly false anti-vaxx information. It was trivial to verify it was wrong, but they claim that they—unlike me—had researched the topic for years. I finally had enough when they were sharing anti-vaxx propaganda that was to “protect the children.”  Another was an old friend who was basically being an apologist for systemic racism. They posted a few quotes from far-right black people in an attempt to justify their viewpoint that there is no systemic racism today, aside from “affirmative action programs”. I’m not going to engage with people like that. It’s too much for me. Racism, whether it’s overt or stemming from a willful disregard for what’s happening in society, is a line in the sand for me.  So I will continue to prune my friend’s list back. Some say that I’m just “staying in my bubble.” That would be true if Facebook was truly a forum for engagement with those who have different points of view. As I’ve found, it’s not.  The Reality  None of the above is surprising to anyone who follows social media: there has been a massive breakdown in social\/political discourse over the past decade. Social psychologist Jonathan Haidt  has spent years researching this and uncovering the root causes. In his article, Why the Past 10 Years of American Life Have Been Uniquely Stupid  , Haidt breaks down the process by which we have grown increasingly polarized. This is largely due to “innovations” in social media. Via “sharing” behavior, coupled with “likes,” we’ve created a system whereby those posts providing provocative content, particularly context which provokes anger, tends to get shared more frequently. People get angry, critical thinking flies out the window, and another angry rant gets a like and a share. We condemn our enemies, flaunt our virtue, and the world gets just a tiny bit darker.  This has been happening millions of times a day, every day, for over a decade and it’s fracturing out society. Truly this is a death by a thousand cuts.  I fully confess to taking part in this, though I’m trying to break out of this cycle. More and more there’s talk of functioning democracy breaking down in the US and perhaps even the US itself breaking apart. Lest that sound alarmist, allow me to quite from the 2022 Texas Republican Party platform  (emphasis mine):   Pursuant to Article 1, Section 1, of the Texas Constitution, the federal government has impaired our right of local self-government. Therefore, federally mandated legislation that infringes upon the 10th Amendment rights of Texas should be ignored, opposed, refused, and nullified. Texas retains the right to secede from the United States , and the Texas Legislature should be called upon to pass a referendum consistent thereto.   Equally ominous:   We reject the certified results of the 2020 Presidential election, and we hold that acting President Joseph Robinette Biden Jr. was not legitimately elected by the people of the United States.   But it’s not just Republicans! 47% of western Democrats have expressed support for breaking away and forming a new country  .  How Do We Fix This?  Enter Jonathan Haidt again. In a 2008 Ted Talk  , Haidt shares the results of his research regarding the moral roots of liberal and conservative thought. He’s found these roots to be shared across the world. The five key roots are:   Ensuring fairness  Caring and compassion  Respecting purity  Respect for authority  Loyalty to your group   Interestingly, while liberals and conservatives generally agree on the first three, liberals tend to reject the last two. Trust in authority must be earned and if the authority abuses that trust, it must be taken away. Group loyalty is also problematic. “America, right or wrong,” is not a liberal ideal.     The conservatives counter, correctly, that while it’s imperfect, authority and loyalty are key to building and maintaining society. Our society works .  The liberal response? It works for you . There are many minority groups in the United States who still do not have equal access to education, jobs, or legal process. Conservatives would prefer that any changes be made slowly, if at all, lest we threaten the fabric of society. Many liberals would push changes through quickly, arguing that we must become a more just society.  So who’s right? Which side should win?  Neither, because thinking in terms of winning or losing is framing the problem incorrectly. Conservatives are correct that authority and loyalty help to buttress society. Liberals are correct that we must evolve society. I tend to be sympathetic to the latter point of view, while acknowledging that it does not always work.  We cannot simply destroy existing institutions without offering credible alternatives. For example, in the United States, much public education is funding via local property, income, and sales taxes. If you live in a poor area, your school is likely to be poorly funded and your education will suffer as a result. We know this is a problem, so how do we fix this?  Well, it turns out we are actively working on solving this, though different states have significantly different approaches, often for very good reasons. In particular, much work is being done to target poor school districts with extra money  . However, that doesn’t actually solve the issue of poverty itself. Simply growing up poor hurts your education outcomes  , starting at kindergarten. Worse, poor children are much more likely to drop out of school, either due to financial or social pressures. We want to fix this, but the fixes themselves are not always clear.  Or take the idea of defunding police . In reality, despite the terrible choice of name, the core idea isn’t fundamentally unsound, but how it’s been viewed (and sometimes promoted) has been terrible. Cut back their budgets substantially to «insert one of numerous explanations here». So if the police forces are smaller and can respond to fewer incidents, what happens when incidents aren’t responded to?  A more reasonable approach is to let the police focus on actual crime and allow social services to handle actual social issues. A mentally-ill homeless man peeing in a doorway isn’t going to get better by being swarmed by gun-toting uniformed officers and then spending time in jail. I have often read police boards where officers lament that they became a police officer to enforce the law, not to be ad-hoc marriage or drug counselors.  If we find a drug addict, stumbling around and acting erratically, do we get them medical help? Do we send them to jail? Do we do both? Should social services respond first, calling the police if they need help? Or should police respond first, ensuring public safety, and then call social services? And if a person clearly needs help, do they lose that right simply because they engaged in the illegal behavior of taking drugs?  Probably everyone one of those answers might be appropriate, depending on the particular situation, but do we have the twin luxuries of money and time to decide every incident on a case-by-case basis?  Instead of conservatives and liberals trying to defeat each other, the sides should look to the unique strengths of the others and cooperate. Liberals should continue to push forward to make the world a better place, while conservatives should be calling them out if their proposals are ill-thought. For example, the state of Oregon recently decriminalized possession of small amounts of cocaine, heroin, LSD, methamphetamine, and other drugs  , but didn’t put the resources in place to provide medical assistance to those in need. Those resources were funded , but the infrastructure didn’t simply didn’t exist, nor were clearly targeted goals outlined. While several countries (most notably, The Netherlands and Portugal) have had very successful decriminalization programs, Oregon is reporting increases in addiction and overdoses  as understaffed hospitals struggle to keep up with the demand. I expect (hope?) they’ll fix this over time, but idealism without pragmatism is a dangerous thing.  The US has the highest incarceration rate in the world  , with almost 1% of the adult population behind bars. As of this writing, there have been 322 mass shootings in the US  , almost two a day, and the number is on the increase. White nationalism is on the rise  , with a 39% increase in hate crimes large cities in 2021  . Teen mental health has declined dramatically across the world  , with much of this being tied to social media consumption.  If we can’t do this for ourselves, can we do this for our children? We need to find the strength to stop sharing negative social media. We need to put forward fact-based information that can be independently verified. If something angers you and you feel “triggered,” perhaps take a break? Just as important, ask yourself if it’s really true. And let’s try to find a way to point fewer fingers, and read the points of view of those you disagree with, especially if they offer solid information to support their point of view.  I don’t know how society is going to be past this. We’re in trouble. We need to stop electing politicians who are little more than walking memes, but I don’t see that happening any time soon. I worry that we won’t be able to put things back together unless they break first.  ","title":"My Facebook Experiment","url":"\/blog\/my-facebook-experiment.html"},{"body":" If you've been in this business long enough, sooner or later you're going to be faced with a terrible problem: fixing the legacy codebase. What follows isn't the only way to proceed, but it's a tried-and-true strategy that unfortunately doesn't appear to be well-known. The core of what follows is risk minimization. Assuming you're facing the problem of fixing a legacy application, you already have risk and you don't need more added to it. What follows is lower risk and lower cost than rewriting the system from scratch.  If you like what you read here and you need help, especially with Perl, get in touch with me and see how our company can help out.  Why You (Probably) Don't Rewrite The Code  Before we start, there are a few things you should know. First, read this now-famous Joel Spolsky article about why you should never rewrite your code (trust me, read it, but don't forget to come back). In that article, Spolsky makes a strong case about why you should refactor your codebase instead of rewriting it. Refactoring, if you're not familiar with the term, is the process of making a series of gradual improvements to code quality without changing the behavior. When you're trying to fix code, trying to change its structure and behavior at the same time is begging for trouble.  That being said, I don't believe in the word \"never\". If your code is written in UniBasic, rewriting might be your only option since you can't find developers who know the language (or are willing to learn it). Heck, I used to program in UniBasic and I've forgotten the language entirely.  Or if you're working with a relatively small piece of software with low impact, rewriting may not be that dangerous.  But let's say you can find or train developers for the language your software is written in, the software is mission-critical, and it's a very large codebase. Rewriting begins to make much less sense. Refactoring it means that you always have working code, you're not throwing away business knowledge or obscure bug-fixes, and your developers aren't starting from scratch, hoping they can make something work. In other words, you're minimizing your risk.  That being said, many companies (and developers) still opt for the rewrite. New code is exciting. New code promises new opportunities. New code is fun but fixing old code is often seen as drudgery. However, if you have a large, legacy codebase, the new code you're writing is, by definition, a large project and large projects are very high risk (emphasis mine):  In a landmark 1995 study, the Standish Group established that only about 17% of IT projects could be considered \"fully successful,\" another 52% were \"challenged\" (they didn't meet budget, quality or time goals) and 30% were \"impaired or failed.\" In a recent update of that study conducted for ComputerWorld, Standish examined 3,555 IT projects between 2003 and 2012 that had labor costs of at least $10 million and found that only 6.4% of [IT projects] were successful .  That's an old study, but there's still plenty of newer work which bears this out. The larger the project, the larger the risk. In fact, of the large projects I have been involved with for various companies, few were both on time and on budget. Some were cancelled outright and still others dragged on, long after it was clear they were a disaster, simply because no one wanted to take the blame for failure. One of them today is approaching its fourth year of a one-year schedule and is riddled with bugs and design flaws, but the company made the new software backwards-incompatible, switched over their clients and now have no way out. The only reason the company is still in business is that they bought another company that is very profitable and is paying for the company's mistake.  That last examples alludes to a dirty little secret that's often not talked about in our industry: large-scale rewrites often exchange one pile of spaghetti code for another. Rather than truly solve the underlying problems, the companies have traded a known set of problems for an unknown set of problems. If you need to fix your legacy code it's because you need to minimize your risk; why on earth would you knowingly adopt unquantifiable risk?  How to Refactor Your Legacy Code  Assuming you've decided that you don't want to face the cost and risk of a large-scale rewrite, how do you refactor your code?  First, you need to assess where you are. At a bare minimum:   What are the functional requirements of the code? (very high-level here)  What documentation exists, if any?  What areas of the code are more fragile? (Check your bug tracker)  What external resources does it require?  What tests exist, if any?   All of these things need to be written down so that anyone can consult this information at a glance. This information represents the bare necessities for the expert you're going to hire to fix the mess. If the above list seems simplistic, that's because we're refactoring, not rewriting.  And yes, you're probably going to hire an outside expert. Not only will they see things that you didn't, but while your current developers may be good, if they can't clearly lay down a solid plan to fix the legacy codebase while simultaneously minimizing risk, you need to bring in someone with experience with this area. What follows is not always intuitive and the expert's experience will help you navigate the rough waters you're already in. At a minimum, your expert needs to have the following:   Expert in the primary language(s) of your codebase  A strong automated testing background  Very comfortable with code coverage tools  A strong database background (probably)  An expert in system design\/architecture  Ability to admit when they're wrong  Understanding of business needs  A persuasive personality   The last points seems strange, but hard choices will need to be made and there will be strong disagreements about how to make them. It's hard to find this mix in top-notch developers, but it will definitely pay off.  Getting Started  The first thing you'll want to do is get a rough idea of how you want your new application laid out. Call this your architecture roadmap, but keep in mind that your landscape will change over time and this roadmap should be flexible. This is where your expert's architecture skills will come in. Various functional parts of your application will be decoupled and put into separate areas to ensure that each part of your application has a \"specialty\" that it focuses on. When each part of your application has one area it focuses on, it's easier to maintain, extend, and reuse, and that's primarily why we want to fix our legacy codebase. However, don't make detailed plans at this time; no battle plan survives first contact with the enemy. Instead, just ensure that you have a rough sense of where you're going to go.  Next, you're going to refactor your application the same way you eat an elephant: one bite (byte?) at a time. You'll pick a small initial target to get familiar with your new tools. Over time, it will get easier, but you don't want to bite off too big a chunk when you get started.  Refactoring a large application means writing tests, but unless you know what you're doing, you're probably going to get it wrong. There's often little TDD here — the code is already written — and you can't write tests for everything — you'll never finish. Instead, you'll be tactically applying integration tests piece by piece.  The first thing you need to do is understand what won't change in your application. By \"won't change\" I mean whatever it is that uses your application's output, whether it be through a JSON API, a Web site, a SOAP interface or what have you. Since something has to use the software, that something is what is going to make everything work. You're going to be writing integration tests against whatever that something is. For the sake of argument, we'll assume we're refactoring a Web application. You've decided that you'll start by writing tests to verify that you can list users on your admin page.  Inside those tests, you'll create a browser object, log in as an admin user, fetch the users page and write tests to assert that the expected users show up on that page. Just getting to this point can often take a huge amount of work. For example, how do you get code to connect to a test database? How do you ensure data isolation between tests (in other words, the order in which tests are run should not matter)? Heck, how do you create that browser object (hint: Selenium is a good choice here)? These and many more questions need to be answered when you're first starting out.  Getting to to this point may be easy if you already have some tests, or it may be very hard if you don't, but it's the important first step in the refactoring.  Once you have that first integration test targeting a small and (relatively) unchanging part of your interface, run your code coverage tools over the test(s) to see what code is covered with these high-level integration tests. Code which is covered is code which is generally safe to refactor (there are plenty of exceptions, but that's another article entirely).  Now you can start looking at which functional parts of the application are embedded in that tested code and make a plan for moving those sections into your architecture roadmap. At this point, it's tempting to rip everything apart, but don't give in to that temptation. Instead, focus on one piece at a time. For example, if you have SQL scattered throughout the code, start pulling that out into your architecture roadmap so that you have a clean API to work with the data you need. Or perhaps you have a Web application and you have been printing the HTML directly: look at using a templating system and start pulling the HTML out into templates. Don't fix everything at once or you'll be trying to do too much. Instead, focus on one area of responsibility and understand it well.  Don't Do Unit Testing (Yet)  Note that we've been talking about integration testing but not unit testing. There's a very good reason for that: with heavy refactoring of a legacy system, your units will change quite heavily when you first start, but the integration tests focusing on the rather static interfaces will not. You want to spend time refactoring your application, not your tests, so until you've stabilized how the code works internally, unit tests can actually be a distraction. Integration testing has the advantage that you can cover (if not actually test) huge portions of your code at once and if done correctly, can be very fast to write. Further, with poorly structured applications, unit testing may be very difficult, if not impossible. Integration testing will also help uncover bugs that unit testing cannot: bugs where different components have different expectations when talking to one another. However, there are some downsides to integration testing:   Integration tests run slower than unit tests  Bugs are harder to track down  It's easier to break real things if you've not isolated your code well enough   That being said, the advantage of integration testing at this stage is clear: refactoring is much easier when you have some basic tests to protect against the worst errors. It's also worth keeping in mind that if you've done little to no testing before this, you're not significantly worse off if you have some solid tests than if you have none. Don't obsess too much on this point: you don't want perfect to be the enemy of the good.  If you haven't already implemented a continuous integration (CI) system, this is the time to start. Even if your developers forget to run the tests, your CI system shouldn't. You want to find out fast if tests are failing.  Pushing Forward  After you've started refactoring one functional piece of a small part of your system, you'll probably quickly uncover some bad assumptions made in the original plan. That's OK. You've started small to minimize your risk. Correct those bad assumptions and then start integration tests with code coverage for another small part of your system, pulling out the functional portions (database calls, HTML, or whatever) that you've already been working on. When you feel comfortable that you've shaken out some of the worst issues, start looking at another functional bit of the system that your currently tested code shares and see if you can pull that out.  Note that this is where your expert's architectural skills are going to shine. They'll understand the importance of decoupling different functional portions of the application. They'll understand how to write robust, flexible interfaces. They'll learn to recognize patterns in your business logic which can be abstracted out. Do not hand this responsibility over to an existing programmer unless you are absolutely confident they have the skills and experience necessary to get this done.  At this point, what follows is a wash\/rinse\/repeat cycle which in worst case scenarios can take years to finish. It takes a long time, but it has some significant advantages:   The code is always working  You're not paying to maintain two systems at the same time  Business knowledge is not lost  New features can still be added  Tests can now be easily written to target existing bugs (even if you don't refactor that code yet)  You can always stop if you've made your codebase \"good enough\"   Why does this approach work? Any large project can seem daunting, but by breaking it down into smaller, manageable pieces, you can at least know where to start and get a sense of where you are going without the nail-biting worry about whether or not a huge project is going to fail.  When I've used this technique before, I've often found that it's a pleasure to finally have a cleaner sense of how the code is evolving and the current experienced team doesn't face the demoralizing prospect of watching their jobs disappear. The downside of this technique is that while code quality improves tremendously, there's always a feeling that it's not good enough. However, as I previously alluded to, many rewritten systems merely create new design flaws to replace old ones. This is far too common of a problem and it means swapping known problems for unknown ones.  Conclusion  The above strategy isn't appealing to many people and it can be a hard sell to those who are convinced that newer is better. In fact, in many respects it can be viewed as boring (though I love refactoring code), but I've successfully used this approach on multiple legacy codebases. However, if you're still trying to decide between a rewrite and a refactor, keep in mind that this approach is a relatively low-cost, low-risk approach. If it proves unworkable, you've likely risked very little. If the rewrite proves unworkable, you could cost the company a ton of money.  So the final question to ask yourself is when you should consider fixing your legacy codebase. The only advice I can offer is to suggest that you not wait until the storm before you fix your roof. Fixing a legacy code base isn't rocket science, but it does require a degree of expert knowledge in how to transform an existing codebase. Sadly, it's not a skill most developers seem interested in acquiring, but then, most don't seem interested in working on legacy codebases in the first place.  ","title":"A Simple Way to Fix Legacy Code","url":"\/articles\/a-simple-way-to-fix-legacy-code.html"},{"body":"    Introduction  Less Boilerplate  param and field  Attribute Shortcuts  Cloning  Customization  Conclusion     Introduction  If you’re waiting for the Corinna OOP project  to land in the Perl core, you’ll have to wait. Paul Evans  will start working on the implementation after Perl version 5.36 is released. However, it’s a lot of work and it’s not clear that if it will be ready by 5.38 and even then, it will be experimental. You’ve a couple of years to wait.  To bridge the gap, I’ve released MooseX::Extended  . It’s not really “Corinna-lite” because the way Moose  works and the way Corinna works aren’t quite compatible, but it will make your Moose code easier to write.  Less Boilerplate  The bane of many programmer’s existence is boilerplate. We hate boilerplate. Not only do we have to type the boilerplate, but we also have to remember it. For a simple, “read only” 2D point class in Moose, the class and my boilerplate looks something like this:  package My::Point::Moose; use v5.20.0; use Moose; use MooseX::StrictConstructor; use Types::Standard qw(Num); use feature qw( signatures postderef postderef_qq ); no warnings qw( experimental::signatures experimental::postderef ); use namespace::autoclean; use mro 'c3'; has [ 'x', 'y' ] =&gt; ( is =&gt; 'ro', isa =&gt; Num ); __PACKAGE__-&gt;meta-&gt;make_immutable; 1;  That’s a lot of boilerplate. Some of it’s not needed for this module, so I am often lazy and don’t write the boilerplate I don’t need, meaning I have to go back and add it later as the class grows.  Here’s the same code with MooseX::Extended :  package My::Point; use MooseX::Extended types =&gt; [qw\/Num\/]; param [ 'x', 'y' ] =&gt; ( isa =&gt; Num );  That’s right, you don’t have to declare it immutable or end it with a true value. MooseX::Extended handles that for you (though you can disable this).  Hmm, what if I want to subclass that and make it mutable?  package My::Point::Mutable::Moose; use v5.20.0; use Moose; extends 'My::Point::Moose'; use MooseX::StrictConstructor; use feature qw( signatures postderef postderef_qq ); no warnings qw( experimental::signatures experimental::postderef ); use namespace::autoclean; use mro 'c3'; has '+x' =&gt; ( is =&gt; 'ro', writer =&gt; 'set_x', clearer =&gt; 'clear_x', default =&gt; 0, ); has '+y' =&gt; ( is =&gt; 'ro', writer =&gt; 'set_y', clearer =&gt; 'clear_y', default =&gt; 0, ); sub invert ($self) { my ( $x, $y ) = ( $self-&gt;x, $self-&gt;y ); $self-&gt;set_x($y); $self-&gt;set_y($x); } __PACKAGE__-&gt;meta-&gt;make_immutable; 1;  Here it is in MooseX::Extended :  package My::Point::Mutable; use MooseX::Extended; extends 'My::Point'; param [ '+x', '+y' ] =&gt; ( writer =&gt; 1, clearer =&gt; 1, default =&gt; 0 ); sub invert ($self) { my ( $x, $y ) = ( $self-&gt;x, $self-&gt;y ); $self-&gt;set_x($y); $self-&gt;set_y($x); }  Again, it lets me focus on writing code that works , rather than all the scaffolding that’s usually needed.  Note: we also provide MooseX::Extended::Role  which behaves similarly.  param and field  You can still use the has function to declare your Moose attributes, but I recommend using param and field instead. Why? We’ve regularly face the following problem:  package Some::Class; use Moose; has name =&gt; (...); has uuid =&gt; (...); has id =&gt; (...); has backlog =&gt; (...); has auth =&gt; (...); has username =&gt; (...); has password =&gt; (...); has cache =&gt; (...); has this =&gt; (...); has that =&gt; (...);  Which of those should be passed to the constructor and which should not? Just because you can pass something to the constructor doesn’t mean you should . Unfortunately, Moose defaults to “opt-out” rather than “opt-in” for constructor arguments. This makes it really easy to build objects, but means that you can pass unexpected things to the constructor and it won’t always work the way you want it to.  There’s an arcane init_arg =&gt; undef pair to pass to each to say “this cannot be set via the constructor,” but many developers are either unaware of this is simply forget about it. MooseX::Extended solves with by separating has into param (allowed in the constructor, but you can also use default or builder ) and field , which is forbidden in the constructor.  We can rewrite the above as this:  package Some::Class; use MooseX::Extended; param name =&gt; (...); param backlog =&gt; (...); param auth =&gt; (...); param username =&gt; (...); param password =&gt; (...); field cache =&gt; (...); field this =&gt; (...); field that =&gt; (...); field uuid =&gt; (...); field id =&gt; (...);  And now you can instantly see what is and is not intended to be allowed in the constructor.  There’s a lot more I can say about that, but you can read the manual  for more information.  Attribute Shortcuts  When using field or param , we have some attribute shortcuts:  param name =&gt; ( isa =&gt; NonEmptyStr, writer =&gt; 1, # set_name reader =&gt; 1, # get_name predicate =&gt; 1, # has_name clearer =&gt; 1, # clear_name builder =&gt; 1, # _build_name ); sub _build_name ($self) { ... }  These can also be used when you pass an array reference to the function:  package Point { use MooseX::Extended types =&gt; ['Int']; param [ 'x', 'y' ] =&gt; ( isa =&gt; Int, clearer =&gt; 1, # clear_x and clear_y available default =&gt; 0, ); }  Note that these are shortcuts and they make attributes easier to write and more consistent. However, you can still use full names:  field authz_delegate =&gt; ( builder =&gt; '_build_my_darned_authz_delegate', );  Cloning  MooseX::Extended offers optional, EXPERIMENTAL support for attribute cloning, but differently from how we see it typically done. You can just pass the clone =&gt; 1 argument to your attribute and it will be clone with Storable ‘s dclone function every time you read or write that attribute, it will be cloned if it’s a reference, ensuring that your object is effectively immutable.  If you prefer, you can also pass a code reference or the name of a method you will use to clone the object. Each will receive three arguments: $self, $attribute_name, $value_to_clone . Here’s a full example, taken from our test suite.  package My::Event; use MooseX::Extended types =&gt; [qw(NonEmptyStr HashRef InstanceOf)]; param name =&gt; ( isa =&gt; NonEmptyStr ); param payload =&gt; ( isa =&gt; HashRef, clone =&gt; 1, # uses Storable::dclone writer =&gt; 1, ); param start_date =&gt; ( isa =&gt; InstanceOf ['DateTime'], clone =&gt; sub ( $self, $name, $value ) { return $value-&gt;clone; }, ); param end_date =&gt; ( isa =&gt; InstanceOf ['DateTime'], clone =&gt; '_clone_end_date', ); sub _clone_end_date ( $self, $name, $value ) { return $value-&gt;clone; } sub BUILD ( $self, @ ) { if ( $self-&gt;end_date &lt; $self-&gt;start_date ) { croak(&quot;End date must not be before start date&quot;); } }  Customization  Not everyone likes MooseX::StrictConstructor . It’s useful to prevent this error:  my $name = My::Name-&gt;new( naem =&gt; 'Ovid' );  In bog-standard Moose, the misspelled attribute will be discarded, leading to bugs that can be hard to track down. However, some code (such as DBIx::Class  ), breaks with a strict constructor. You can skip that:  use MooseX::Extended excludes =&gt; [qw\/StrictConstructor\/];  Both MooseX::Extended and MooseX::Extended::Role have a variety of features you can exclude if they don’t work for you. Again, see the documentation.  Note that at the present time, this customization is per class and cannot be inherited. This may change in a future release.  Conclusion  There’s a lot more packed into MooseX::Extended , but this covers the highlights. I think MooseX::Extended will make many Perl programmer’s life easier if they use Moose, but want a better set of defaults. It’s still in ALPHA, but it' close to a feature-complete BETA release. I think it’s relatively solid, but no guarantees. The github repo  has Linux and Windows CI workflows set up to catch errors quickly and the test suite is coming along nicely.  If you’re interested in starting any new projects using Moose, please give MooseX::Extended a try and let me know what you think.  ","title":"Introducing MooseX::Extended for Perl","url":"\/articles\/introducing-moosexextended-for-perl.html"},{"body":"    Introduction  Programming in the Wrong Language  I Failed at Corinna  What I Was Trying to Do  How I Fixed It  Types (or the lack thereof)  Exceptions  Conclusion     Introduction  This post is mainly my findings on finally porting some real code to Corinna, but I confess I was concerned when I first started.  I recently tried to convert a bless object hierarchy to Corinna  and I failed, badly. The code worked, but the design was a mess. As the lead designer of Corinna, after years of effort, I found that Corinna was flawed. After hours of trying to write and rewrite the explanation of the flaw, I found the flaw: me.  Corinna is not flawed. I was thinking in terms of blessed hashes, not Corinna. When I realized what was going on, my thoughts shifted. I started to see the old code was riddled with flaws. Admittedly, the code I was porting was something I designed (*cough*) two decades ago, but it was still humbling.  Programming in the Wrong Language  It used to be that plenty of C developers flocked to the Perl language because it’s so quick-n-easy to whip up a prototype and test out an idea. Of course, experienced Perl programmers would point out that the C programmers were writing Perl as if it was C, not Perl. They’d look at this:  for ( my $i = 0; $i &lt;= @array; $i++ ) { my $element = $array[$i]; ... }  By the way, did you spot the bug?  And they’d say, “just use the elements directly”:  for my $element (@array) { ... }  That easier to read, more likely to be correct, and it’s faster.  In the 90s, I didn’t have that problem when I started with Perl. At the time, it had been over a decade since I had done any serious C hacking, so I didn’t write my Perl like C. I wrote it like COBOL. Seriously! I’d declare all of my variables at the top of the program, as if I were setting up my WORKING-STORAGE SECTION  . Then I’d set the appropriate variables and call the procedure subroutine I needed, though I at least returned data instead of setting a global.  Needless to say, my early days in writing Perl were pretty awful, but it was still a hell of a lot easier than COBOL.  Today, many Perl developers are excited about the Corinna project, for which I have written a short tutorial.  Just as one should stop thinking in C or COBOL when writing Perl, one should stop thinking in terms of using bless  when writing Corinna. If that seems like it’s not too hard, I can assure you, many will stumble as I did.  I Failed at Corinna  My HTML::TokeParser::Simple  module is modestly popular. There are over 30 distributions which depend on it  and I frequently see it used in clients I assist, and given that it’s been around for over two decades with no bugs reported, I’m pretty happy with that module. So when Paul Evans created a PR for a subset of Corinna  , I thought it was time to port something. Instead of mocked-up code in the RFC  or my experiments with Object::Pad  , I was going to write real Corinna code.  I had been following along and giving Paul some feedback on development work. I found a couple of bugs (not many, which is impressive), but now it was time to really push things. Hell, as the lead designer of the Corinna project, based on my original 2019 (mis)design  and that was based on research and work I had done prior to that , I’m pretty comfortable with saying that if anyone can port something to Corinna, I am that guy.     And that’s when I discovered that I’m Prax, not Amos. I am not that guy. (If you haven’t watched or read The Expanse, you’re missing out. Trust me.)  What I Was Trying to Do  The point of this module is that HTML::TokeParser  could parse HTML into a stream of tokens which look like this:  [&quot;S&quot;, $tag, \\%attr, \\@attrseq, $text] [&quot;E&quot;, $tag, $text] [&quot;T&quot;, $text, $is_data] [&quot;PI&quot;, $token0, $text] [&quot;C&quot;, $text] [&quot;D&quot;, $text]  The code I wrote using that was constantly breaking, so I blessed those and put an OO wrapper on them so that $token-&gt;as_is (renamed to $token-&gt;to_string in my new code) always returned what it was supposed to return, instead of using $token-&gt;[4] (start tag), $token-&gt;[2] (end tag), $token-&gt;[1] (text), an so on. You can’t even use $token-&gt;[-1] to read the last item thanks to the T token (text) ironically being the token which had the plaintext in an unpredictable position.  That’s much easier than using HTML::TokeParser directly. I did this by calling bless with the array references and blessing them into appropriate classes. This meant the array reference was still available and HTML::TokeParser::Simple was a drop-in replacement for the original module. You could switch from HTML::TokeParser to HTML::TokeParser::Simple with no other changes in your code. You then gradually converted array reference lookups to method calls. I was doing a lot of web scraping in the pre-API days of the Web and this saved me much grief.  So when I started designing HTML::TokeParser::Corinna  , I hit my first snag.  Since Corinna is designed to be encapsulated, you can’t call $token-&gt;[1] . No “reaching inside” the object is allowed. But that’s fine! Since HTML::TokeParser::Corinna is a new module, I can create any interface I want. That’s when I hit my next problem.  For each of the array reference types listed above, I have a corresponding class:   HTML::TokeParser::Corinna::Token::Tag::Start  HTML::TokeParser::Corinna::Token::Tag::End  HTML::TokeParser::Corinna::Token::Text  HTML::TokeParser::Corinna::Token::Comment  HTML::TokeParser::Corinna::Token::Declaration  HTML::TokeParser::Corinna::Token::ProcessInstruction   There are some common behaviors there and since we don’t yet have roles for Corinna, I used abstract base classes. (We’ll shorten the prefix to the namespace to make it easier to read):   HTC::Token  HTC::Token::Tag :isa(HTC::Token)   I can instantiate a corresponding class like this, with all constructors having the same interface:  my $end_tag = HTC::Token::Tag::End-&gt;new( token =&gt; $token );  Since HTC::Token is the base class for everything, I have this:  class HTC::Token { field $token :param; method _get_token () {$token} ... }  It also has the methods common to all token classes.  Subclasses look like this:  class HTC::Token::Tag :isa(HTC::Token) { ... } class HTC::Token::Tag::Start :isa(HTC::Token::Tag) { ... }  Even ignoring the fact that my objects were mutable, my code is flawed. The “Tag” classes need to be able to access the $token from the parent class. I have no way to do that, so I have a _get_token method. Untrusted code can call $token-&gt;_get_token and change the array reference in unexpected ways. That kills one of the major points of Corinna, but I’ve no easy way of sharing that data otherwise.  Realizing I could not fix this was my crushing blow, leading me to naïvely believe Corinna was flawed. What follows is how I worked through the issue, but it took longer for me to have clarity than what is shown here.  How I Fixed It  One way of handling this is the following:  class HTC::Token { field $token :param; method _get_token () {clone($token)} ... }  But that still leaves _get_token() callable outside the class and it’s now part of the interface. It becomes an implementation detail I don’t have the freedom to change (classes should be open for extension, not modification). It’s part of the class contract and should not be violated.  Corinna doesn’t have a clean way of handling this case, but it’s not a flaw. It’s a limitation and one we can easily fix. Adding a :trusted attribute  to methods would make this much easier, but that’s still an open discussion.   A trusted method, whether provided by an abstract base class or a role, should propagate to the first non-abstract subclass and become a private method in that class. If it’s defined directly in a concrete (non-abstract) class, then the first concrete class which inherits it gains it as a private method.   This isn’t quite how trusted methods work in other languages, but that’s OK. Perl is not like other languages and we have to adapt.  Lacking trusted methods, I cut-n-pasted the field $token :param; line into each of my concrete classes. That solved the problem, but if they each took multiple parameters, having to sync them across multiple classes would be fragile. A single role (or base class) providing those as :trusted would make this issue go away.  So, bullet dodged. Corinna isn’t irrevocably broken, but it did give me a bit of a scare at first. However, it also pleased me. “No plan survives first contact with the enemy,” but I confess I had a small concern. Despite years of research and design, maybe we had missed something critical. Finding only a tiny limitation has been a relief (though whether this holds remains to be seen).  Types (or the lack thereof)  This next part isn’t about a limitation of Corinna, but of my not understanding object-oriented design when I first wrote HTML::TokeParser::Simple . This is related to type theory.  A type system is nothing more than a way of evaluating expressions to ensure they do not produce unwanted behavior. Perl’s primary type system is based on data structures, not data types .   For example, you can’t access an array element the way you access a hash element (though Perl being Perl, there are ways you can change that, too). But what is two plus two? We know that the answer is four, yet the computer often needs help. Let’s look at the following:  my @array = qw(10 11 12); my $var = \\@array; say 2 + 2; # Int + Int say 2 + &quot;2&quot;; # Int + String say &quot;2 pears&quot; + &quot;2 weeks&quot;; # Str + Str say 2 + @array; # Int + Int! say 2 + $var; # Int + Ref  That prints something like:  4 4 Argument &quot;2 weeks&quot; isn't numeric in addition (+) at two.pl line 8. Argument &quot;2 pears&quot; isn't numeric in addition (+) at two.pl line 8. 4 5 5201037554  For many languages, only 2 + 2 would be valid in the above. Perl is heavily optimized for text manipulation, so if you’re reading in a bunch of data from a text file, you can often treat numeric strings as numbers. Thus, 2 + \"2\" is 4 . The ASCII value of \"2\" is 50, but Perl understands what you mean and casts the string as an integer instead of calculating 2 + 50 .  The \"2 pears\" + \"2 weeks\" is clearly nonsense, but at least you get a warning.  2 + @array surprises many people new to Perl, but it’s evaluating @array in scalar context. Since it has three elements, this reduces to 2 + 3 , printing 5 . I know several programmers who write this as 2 + scalar @array to be explicit about the intent.  But what’s with that 5201037554 in the output? Your number will vary if you run this code, but what’s happening is that $var , in the expression 2 + $var , evaluates to the address of the reference \\@array . You don’t even get a warning. This is useless (no pointer math in Perl) and yes, I’ve been bitten by this in production code.  For many languages this expression would prevent your program from compiling, but Perl is Perl. For the poor maintenance programmer seeing my $result = $var1 + $var2; buried deep in the code, it may not be immediately obvious there’s an issue.  So this gets us back to a fundamental question: what is a type? A type is nothing more than:   A name for the type  A set of allowed values for that type  A set of operations allowed to be called on that type   If we think of an integer as an object and addition as a method, let’s play with some pseudocode and pretend we have multimethods and a way of declaring data types.  class Int { field $int :isa(Int); multimethod add ($value :isa(Int)) { return $int + $value; } multimethod add ($value :isa(Str) :coerce(Int)) { return $int + $value; } } my $int = Int-&gt;new( int =&gt; 2 ); say $int-&gt;add(3); # 5 say $int-&gt;add(&quot;4&quot;); # 6 # runtime error because we can't coerce say $int-&gt;add(&quot;4 apples&quot;); # problematic because arrays flatten to lists and # an array with one element will work here, but # zero or two or more elements are fatal say $int-&gt;add(@array); # fatal because there is no multimethod dispatch target say $int-&gt;add(\\@array);  In the above, we simply don’t provide methods for behaviors we don’t want. Yes, the developer may very well have to check that they’re not passing in bad data, but this is not a bad thing. At their core, objects are experts about a problem domain and you need to take care to get them right.  This also fits with the principle that we want to minimize our interfaces as much as much as possible. The more methods you expose, the more methods you have to maintain. If you later need to change those methods, you may break existing code. So let’s look at my abstract HTC::Token base class, a more-or-less straight port of the original code:  class HTML::TokeParser::Corinna::Token { field $token : param; method to_string { $token-&gt;[1] } method _get_token {$token} method is_tag {false} method is_start_tag {false} method is_end_tag {false} method is_text {false} method is_comment {false} method is_declaration {false} method is_pi {false} method is_process_instruction {false} method rewrite_tag { } method delete_attr { } method set_attr { } method tag { } method attr (@) { {} } method attrseq { [] } method token0 { } }  That ensures that every class has a stub of every method available to it, so you won’t get a “method not found” error. But what if you have a token representing text in your HTML? Why on earth would you want to call $token-&gt;rewrite_tag if it’s not a tag? It’s like the above example of adding an integer to a reference: you can do it, but it’s not helpful.  What is helpful is knowing what kind of token you have. So my base class is now:  class HTML::TokeParser::Corinna::Token { method is_tag {false} method is_start_tag {false} method is_end_tag {false} method is_text {false} method is_comment {false} method is_declaration {false} method is_pi {false} method is_process_instruction {false} }  This is cleaner and easier to maintain. In fact, I could delete this class, but those predicate methods are much easier to use.  if ( $token isa 'HTC::Token::Tag::Start' ) { ... } # versus if ( $token-&gt;is_start_tag ) { ... }  I’ve also taken the trouble to make all tokens immutable. We generally want immutable objects , but in reality, sometimes it’s cumbersome. If you want to replace the class newz with news everywhere, here’s what you do:  my $parser = HTC-&gt;new( file =&gt; $file ); while ( my $token = $parser-&gt;next ) { if ( $token-&gt;is_start_tag &amp;&amp; $token-&gt;attr('class') eq 'newz' ) { $token = $token-&gt;set_attrs( class =&gt; 'news' ); } print $token-&gt;to_string; }  The mutators such as set_attrs now return a new instance instead of mutating the token directly. That makes it safer because you don’t worry about unrelated code mutating your data. For example, if you call $object-&gt;munge(3) , you never worry that the value of 3 has suddenly changed in your code. However, $object-&gt;munge($other_object) offers no such guarantee.  In the code snippet above, however, always having to remember to assign the return value feels, well, clumsy. In fact, if you call set_attrs in void context (i.e., you don’t assign the return value to anything), the code will throw a HTML::TokeParser::Corinna::Exception::VoidContext exception (yes, it now has true exceptions, but they’re part of this module, not part of Corinna).  So my interfaces are smaller and we no longer provide useless, potentially confusing methods. I think that’s a win.  Exceptions  As a final note, I was proud of an odd little trick. I wanted to use exceptions as much as possible. They fix a very common bug in production code. If someone calls die or croak , you often see code like this:  try { some_code(); } catch ($e) { if ( $e =~ \/connection gone away\/ ) { # retry connection or rethrow exception } die $e; };  Now if some maintenance programmer renames the error message to connection interrupted , all code dependent on the previous error message breaks. But if they throw an exception in an Exception::Connection::Lost class, the code can check the class of the exception and the developers are free to change the actual error message any way they like.  So here’s my exception base class:  class HTML::TokeParser::Corinna::Exception { use overload '&quot;&quot;' =&gt; 'to_string', fallback =&gt; 1; use Devel::StackTrace; field $message :param = undef; field $trace = Devel::StackTrace-&gt;new-&gt;as_string; method error () {&quot;An unexpected error occurred&quot;} method message () {$message} method stack_trace () {$trace} method to_string { # error() can be overridden my $error = $self-&gt;error; # but $message is universal if ($message) { $error .= &quot;\\n$message&quot;; } return &quot;Error: $error\\n\\nStack Trace:\\n$trace&quot;; } }  Because stringification is overloaded, I can print the exception or check it with a regex. Because it’s an object, you can check the class of the exception to decide what to do next.  I used the above to provide a MethodNotFound exception:  class HTC::Exception::MethodNotFound :isa(HTC::Exception) { field $method :param; field $class :param; method error () { &quot;No such method '$method' for class '$class'&quot; } method method () {$method} method class () {$class} }  And in my base class, I have this:  method AUTOLOAD { our $AUTOLOAD; my ( $class, $method ) = ( $AUTOLOAD =~ \/^(.*)::(.*)$\/ ); return if $method eq 'DESTROY'; throw( 'MethodNotFound', method =&gt; $method, class =&gt; $class, ); }  And now, $token-&gt;no_such_method throws an exception instead of causing you to die inside.  Conclusion  The earlier description of the hours of writing and rewriting to explain the flaw encompass much more than what I’ve discussed, but I wanted to keep this short. Of course, I threw in a few other things I noticed along the way.  The encapsulation violation seemed to break the main strength of Corinna, but spending a few hours porting a class hierarchy quickly exposed the design limitation and a solution presented itself. Perhaps the Corinna design team or someone else might offer a more elegant solution than what I presented. I’m OK with that.  So far, the Corinna code is simpler, easier to read, provides strong encapsulation, and was generally a pleasure to write. I’m looking forward to the day it’s production ready. I expect there will be teething pain for others, since thinking in terms of blessed hashes is ingrained in Perl culture, but if we keep living in the past, Perl will become a thing of the past.  The first PR for use feature 'class' is here  and I’m sure any and all feedback would be appreciated.  ","title":"Converting Object-Oriented Code to Corinna","url":"\/articles\/converting-object-oriented-code-to-corinna.html"},{"body":"    Quickstart  The Overview  git-flow  Our git Workflow  Assumptions  Checking Out a Branch  Caveats  Refreshing Your Branch  Merging into master  Determining Your Primary Branch  Merge Conflicts  Update        We say what we do and do what we say.    Note : For historical reasons, we have left the name master in a number of places in this document. Many projects prefer that this name be main or something different.   One of the things we constantly strive to do at All Around the World  is apply the Pareto Principle  : 80% of your results stem from 20% of your actions. When we go in to solve problems for clients; we don’t have the luxury of sitting around for days to debate the perfect way of doing things before we start working. Instead, we strive to find the 20% of actions which give us 80% of the results and uncover the rest as we go. Today, I’ll talk about our git workflow  and how three simple commands make our team collaboration much easier. Email us to discuss how we can build a remote software team for you. A team that delivers, not debates.  Quickstart  Using our git workflow couldn’t be simpler:  (master) $ git hub 123 # create new branch (new-feature-123) $ # hack, hack, hack (new-feature-123) $ git refresh # pull in latest master changes (new-feature-123) $ # hack, hack, hack (new-feature-123) $ git done (master) $ # done  And that’s it. Three simple commands to remember and git gets out of your way .  The Overview  The git command is extremely complicated. Invariably in a team environment someone gets something “wrong” with git and the local git guru comes by and does mysterious things with git fsck , git reflog , and other commands that developers have never heard of. And while it’s useful to have a git guru on hand, we find it’s even easier to have a single, standard way of using git so that all developers are doing things the same way. Even developers new to git find our workflow very easy.  It’s designed largely to avoid this:    A typical git history.   For us, the complicated history was the showstopper. We want a clean, linear history and an “easy-to-use” workflow. It’s based on the following logic:   The master branch is the source of truth  All new development happens in a new branch off of master  When you’re done, merge your code back into master and delete your branches   In other words, it’s very clean, very simple, and focuses on the core development needs. There’s nothing surprising about this workflow. How do releases work? That depends on your project and its needs. How do hotfixes work? That depends on your project and its needs. How do code reviews work? That depends on your project and its needs. Like the WildAgile project management system , our workflow is designed to be the core of what you do day in and day out. The bits you need to customize are up to you. We’re not dogmatic.  git-flow  As an aside, one popular workflow is git-flow  . Yet git-flow has its detractors  and it’s easy to find many more  . We found the convoluted history and strict requirements that things be “just so” to not be a good fit for our workflow. It might be the perfect fit for yours.  Our git Workflow  The git workflow tools  are open source and they contain a simplified subset of the git tools used by All Around the World  for our software development. It makes it dead easy for teams using git (and in our case, github) to work together.  There are only three new commands to remember:   git hub $issue  Create new branch based on a github ticket  git refresh  Rebase your work on top of current master  git done  Merge your branch back into master   Note: only the bin\/git-hub command assumes you’re using github. The other commands work fine without it.  Assumptions  The master branch is never worked on directly. Instead, new branches are created (usually for an individual github ticket), we hack, we regularly pull new changes into that branch, and after a pull request is created and the approved, we merge the code back into master .  The examples below assume the files in the bin\/ directory are in your path. If they are not in your path, you have to type the commands explicitly:  bin\/git-hub 5738 bin\/git-refresh bin\/git-done  Checking Out a Branch  We assume that branches are usually per ticket. You can manually create a branch, but we tend not to. Instead, if you are going to work on github issue 5738, with the title “Lower reputation for failed crimes”, you run the following command:  git hub 5738  And you’re in a shiny new branch named lower-reputation-for-failed-crimes-5738 .  If that branch already exists, it’s checked out.  If you’re not using github, it’s trivial to do this by hand:  git checkout -b name-of-your-branch  We just like the convenience of always having standard naming conventions.  Caveats   The new branch that is created is based on the branch you’re on, so you probably want to run this from master  Branch detection is based on a heuristic, using the ticket number. If you have more than one branch with that ticket number, you will get a warning and the code will exit.  You’ll need a config file for this. See perldoc bin\/git-hub for this.  Assumes Perl 5 version 20   You will also need to install the following modules from the CPAN  if you’ve not already done so.   autodie  Config::General  JSON  Text::Unidecode  Pithub   If you’re not familiar with installing CPAN modules, check out cpanminus  , or you can use the cpan command line tool that is standard with Perl.  Refreshing Your Branch  While working on your branch, you want to regularly pull in the latest master to keep your code up-to-date. Working on a change for a week with a fast-moving codebase can cause serious headaches when it’s time to merge. Thus, you should regularly run the following in your branch (I try to run it at least once per day):  git refresh  Regardless of the branch you are on, this code:   Stashes changes (if any)  Checks out master  Does a fast-forward merge  Checks out your branch (if branch is not master)  Rebases onto master (if branch is not master)  Pops changes from stash, if any   In other words, it cleanly rebases your code on top of master, even if you have uncommitted changes.  Note: for everyone who keeps asking “why not git pull --rebase --autostash ?”  The simple reason is backwards-compatibility. Sometimes developers have older versions of git installed and while that’s usually perfectly compatible with newer versions, we don’t force them to upgrade. Also, internally git-refresh does things like git fetch -p to remove references to remote branches which no longer exist. This regular “house cleaning” helps to keep git more performant. See the Pruning  documentation for git-fetch for more information.  Note: if this command aborts (such as during a merge conflict), the stashed changes will remain stashed. Run git stash to see those changes and git stash pop to return them to your codebase when you’ve finished resolving the merge conflicts.  Merging into master  So you’ve finished your work and are ready to merge your branch back into master. Here’s one way to do that very cleanly and safely:  git checkout master git fetch --prune git merge --ff-only origin\/master git rebase master my-awesome-feature-1234 git push --force-with-lease origin my-awesome-feature-1234 git checkout master git merge --no-ff my-awesome-feature-1234 git push origin master  You’ll remember that, right? And you’ll never mess it up?  For us, we simply run:  git done  And that will cleanly update master (or whatever the source branch is) and rebase your branch on top of it, and push that change to your origin.  With that, you get a clean git history like this:  | * 44eba1b094 - ... | * 217350810f - ... |\/ * c84e694e59 - Merge branch 'no-add-message-from-context-object-8615' PR#8622 (6 days ago) &lt;some author&gt; |\\ | * 9d73143f75 - ... | * 983a1a5020 - ... | * e799ecc8e3 - ... | * aa9c981c2e - ... | * 4651830fd6 - ... |\/ * 010e94b446 - Merge branch 'fix-test-combat-module-usage' PR#8623 (7 days ago) &lt;some author&gt; |\\ | * 46d8917af7 - ... |\/ * 4acfdd8309 - Merge branch 'economy-action-use-args-hashref' PR#8617 (10 days ago) &lt;some author&gt; |\\ | * a1f863f908 - ... | * b3dc3efb2a - ... | * ab77373fca - ... | * b5491e4ae9 - ...  And when you’re done, it will also print out the instructions on how you can delete your local and remote copies of the branch, to help you be nice and not leave branches cluttering up the repository.  Determining Your Primary Branch  Internally the code attempts to detect the name of your primary branch using this:  basename $( git symbolic-ref refs\/remotes\/origin\/HEAD )  That will usually work. For all commands, if it does not work, it will print out the following message:  Could not determine target branch via '&quot;basename $( git symbolic-ref refs\/remotes\/origin\/HEAD )&quot;' You scan set your target branch with: git symbolic-ref refs\/remotes\/origin\/HEAD refs\/remotes\/origin\/\\$branch_name Where $branch_name is the name of the primary branch you develop from ('main, 'master', etc.)  Merge Conflicts  With all of this rebasing, you’re sometimes going to hit the dreaded ...  Auto-merging lib\/Veure\/Moose.pm CONFLICT (content): Merge conflict in lib\/Veure\/Moose.pm Failed to merge in the changes. Patch failed at 0001 Issue 144 Make read-only the default for attributes  And then you’re very unhappy. Worse, if you do this after a week of isolated hacking, you might get a huge amount of merge conflicts to work through. However, by running git refresh at least once a day, if not more often, those conflicts are minimized. And when you do get them? I have the following in my .bash_aliases file:  alias damnit='vim $(git grep -l \"&lt;&lt;&lt;&lt; HEAD\")'  Then, when I get a conflict, I just type damnit and am automatically editing the files which caused the conflict. It saves me a huge amount of time.  Update  We have renamed git-merge-with-master to git-done . The former still works.  Check out the tools at our github repo  .  ","title":"Easy Git Workflow","url":"\/articles\/easy-git-workflow.html"},{"body":"  When I was working with the BBC, the age-old debate had arisen again: should we use “cuddled elses” or not?   For those not familiar with the debate, it’s asking if we should do this:  if (someCondition) { System.out.println(&quot;We all gonna die&quot;); } else { System.out.println(&quot;We not gonna die&quot;); }  Or this:  if (someCondition) { System.out.println(&quot;We all gonna die&quot;); } else { System.out.println(&quot;We not gonna die&quot;); }  For those of you who are not software developers, the differences might seem pedantic. For those of you who are software developers, the difference is pedantic. Unfortunately, it caused a furious debate (as it often does) at the BBC and, as often happens with these debates, work isn’t done. Sometimes tempers flare. And quite often a decision isn’t made.  And that’s actually bad because one developer might “cuddle” that else and another developer working on this code might “uncuddle” it. And they’re not being malicious or petty: “correctly” formatting code is simply something an experienced developer does as part of their job. And that means more commits to your version control system and more work digging through those commits if you’re trying to figure out why something was written in a particular way.  Or worse, you have a developer who decides to write code to format the entire codebase in their personal style and running git annotate $filename shows that developer on all the lines. So you contact this “expert” about the $filename and they say they’ve never heard of it. I’ve run across this mess more than once.  Getting back to the BBC ...  We had agreed we’d use Perl::Tidy  to automatically format our code, but at that point, we hit another obstacle because everyone argued about the brace style (if you don’t know what that means, trust me, you don’t care) and no matter how hard we tried, Perl::Tidy couldn’t get the curly braces formatted the way we wanted them to be formatted.  My argument was “who cares? We’ll get used to the new styles. Let’s get back to work.” But another developer, passionate about his brace placement, argued that he could write a PPI  hook to fix this. Sigh.  I quietly bet another developer £5 that this approach would not work. I’m sad to say I won that bet. Worse, when the “solution” was ready, it kept hitting edge cases in our code and breaking.  And if you really want to see what a ridiculous timesink this can be, you can choose many different indentation styles such as K&amp;R (and its many variants), Allman, Pico, Ratliff, GNU, ad nauseum   What a waste of time!  The icing on the cake? Though we know having regular formatting of code to show the logical structure of the program makes it easier to follow the code (proof: read minimized Javascript versus unminimized), there’s no strong evidence (that I’ve ever found) a particular type of formatting is better than another. If you disagree, please provide non-anecdotal evidence.  In fact, in the excellent book Code Complete: A Practical Handbook of Software Construction, Second Edition  , by Steve McConnell, he dives into this at the beginning of chapter 31. After some discussion of the topic, including referencing some studies on said topic, he concludes (emphasis mine):   Expert programmers often cling to their own styles tenaciously, even when they’re vastly different from other styles used by other expert programmers. The bottom line is that the details of a specific method of structuring a program are much less important than the fact that the program is structured consistently .   Unsurprisingly, these arguments about brace style, cuddled elses, tabs versus spaces,   and so on, are almost meaningless. To be fair, there is marginal value to be had in suggesting things such as “shorter variable names for smaller scopes” and “indent to show logical structure”, but being too pedantic about it pisses people off for no good reason.  So how do we fix this?  First, pick a reasonable style you can automate , swallow your pride about the details, and choose your automation tools. Second, consider that—unless you’re pair programming—noone really cares much about the font you’re using, do they? Not really. You’ve set your editor to choose your own font and that’s not written back to your source code, so set your editor to choose your own style and make sure it’s not written back to the source code.  So let’s say you prefer spaces to tabs, with a four-space indent, and you use vim to edit your code. When you open a file, or create a new one, using the BufRead,BufNewFile commands is the first thing you need, or you can create a mapping to apply your style. For example, in your filetype mappings you can add the following:  noremap &lt;silent&gt; &lt;leader&gt;style :call s:ApplyMyStyle()&lt;cr&gt; function! s:ApplyMyStyle() &quot; set up my editor to do the right thing setfiletype perl set expandtab set textwidth=78 set tabstop=4 set shiftwidth=4 set expandtab retab &quot; apply my tab\/space choices &quot; reformat the file execute(&quot;:%!perltidy -q --profile=myperltidyrc&quot;) endfunction  Now, assuming your leader is the backslash, you can type \\style to apply your coding style and never, ever have to pay attention to the silly choices other developers make.  Do I recommend this technique? Absolutely not. You’re going to commit in your own style one day. However, there are git hooks you can write to apply styles when you commit  . It’s much safer. A better choice, still, is to just be an adult and admit your “one true style” is a personal preference, not a mandate handed down from heaven. Learn to use the house style in preference to your own and avoid all of the petty arguments in the first place. You won’t look like a better person for it, but you’ll look like a worse one if you tell the other programmers they’re fools for choosing a style different from your own.  ","title":"Automate Your Software Standards","url":"\/articles\/automated-software-standards.html"},{"body":"  The Problem  Test databases are very easy to get wrong. Very easy. Decades ago when I first learned testing, the team shared a test database. If you ran your test at the same time another developer, both of your test suites would fail! However, we were using a database where we had to pay for individual licenses, so we were limited in what we could do.  Later, I worked for a company using MySQL and I created an elaborate system of triggers to track all database changes. This let me “fake” transactions by starting a test run, see what had changed last time, and automatically reverting those changes. It had the advantage that multiple database handles could see each other’s changes (hard to do for many databases if you have separate transactions). It had the disadvantage of everything else: it was fragile and slow.  Later, I started using xUnit frameworks, eventually writing a new one  that’s popular for companies needing a large-scale testing solution. With this, it was easy for each test class to run in a separate transaction, cleaning itself up as it went. Using transactions provides great isolation, leverages what databases are already good at, and let’s you run many classes in parallel.  But it can easily break embedded transaction logic. And you have to guarantee everything shares the same database handle, and you can’t really test the transactions in your code, and, and, and ...  What finally drove me over the edge was writing some code for a client using the Minion job queue  . The queue is solid, but it creates new database connections, thus ensuring that it can’t see anything in your database transactions. I figured out a (hackish) solution, but I was tired of hackish solutions.  While I was researching the solution, Matt Trout was reminding me (again) why the “database transactions for tests” approach was broken. Just spawn off temporary test databases and use those, throwing them away when you’re done.  The Client  So a company wanting to hire me gave me a technical test and there was a task to add a small, simple feature to a Catalyst web application  . It was trivial. They handed me a Vagrant file and after a quick vagrant up and vagrant ssh , I was ready to begin. Then I looked at the test they had for the controller:  use strict; use warnings; use Test::More; use Catalyst::Test 'Client'; ok( request('\/some_path')-&gt;is_success, 'Request should succeed' ); done_testing();  The task involved a POST to a URL. There was no test for the existing feature that I was adding to, but any test I wrote meant I’d be changing the state of the database. Run the code multiple times and I’d leave junk in the database. There were various ways I could approach this, but I decided it was time to build a quick database on the fly, write to that, and then dispose of it after. The code for this was trivial:  package Test::DB; use File::Temp qw(tempfile); use DBI; use parent 'Exporter'; use Client::Policy; BEGIN { if ( exists $INC{'Client\/Model\/ClientDB.pm'} ) { croak(&quot;You must load Test::DB before Client::Model::ClientDB&quot;); } } use Client::Model::ClientDB; our @EXPORT_OK = qw(test_dbh); my $connect_info = Client::Model::ClientDB-&gt;config-&gt;{connect_info}; my $dsn = $connect_info-&gt;{dsn}; my $user = $connect_info-&gt;{user}; my $password = $connect_info-&gt;{password}; # $$ is the process id (PID) my $db_name = sprintf 'test_db_%d_%d', time, $$; my ( $fh, $filename ) = tempfile(); my $dbh = DBI-&gt;connect( $dsn, $user, $password, { RaiseError =&gt; 1, AutoCommit =&gt; 1 } ); $dbh-&gt;do(&quot;CREATE DATABASE $db_name&quot;); system(&quot;mysqldump -u $user --password=$password test_db &gt; $filename&quot;) == 0 or croak(&quot;mysqldump failed: $?&quot;); system(&quot;mysql -u $user --password=$password $db_name &lt; $filename&quot;) == 0 or croak(&quot;importing schema to mysql failed: $?&quot;); # XXX We’re being naughty in this quick hack. We’re writing # this back to the Model so that modules which use this connect # to the correct database. $connect_info-&gt;{dsn} = &quot;dbi:mysql:$db_name&quot;; # This is just a quick hack to get tests working for this code. # A catastrophic failure in the test means this might not get # run and we have a junk test database lying around. # Obviously we want something far more robust END { $dbh-&gt;do(&quot;DROP DATABASE $db_name&quot;) } sub test_dbh () { $dbh } 1;  The above is very naughty in many ways, but the client hinted that how fast I returned the test might be a factor (or maybe they didn’t and I misread the signals). They also made it clear they were looking at how I approached problems, not whether or not the code was perfect. Thus, I thought I was on safe territory. And it meant I could do this in my test:  use strict; use warnings; use Test::More; use lib 't\/lib'; use Test::DB; use Catalyst::Test 'Client'; ok( request('\/some_path')-&gt;is_success, 'Request should succeed' ); # anything I do here is against a temporary test database # and will be discarded when the test finishes done_testing();  The Test::DB code was quick and easy to write and made it trivial for me to safely write tests. I was pleased.  What’s Wrong With Test::DB?  For a junior developer, Test::DB might look awesome. For an experienced developer, it’s terrible. So what would I do to make it closer to production ready?  Here are just a few of the things I would consider.  Stronger Data Validation  First, let’s look at our connection information:  my $connect_info = Client::Model::ClientDB-&gt;config-&gt;{connect_info}; my $dsn = $connect_info-&gt;{dsn}; my $user = $connect_info-&gt;{user}; my $password = $connect_info-&gt;{password};  The above relied on how Catalyst often sets up its DBIx::Class (a Perl ORM) model:  package Client::Model::ClientDB; use strict; use base 'Catalyst::Model::DBIC::Schema'; __PACKAGE__-&gt;config( schema_class =&gt; 'Client::Schema::ClientDB', connect_info =&gt; { dsn =&gt; 'dbi:mysql:test_db', user =&gt; 'root', password =&gt; 'rootpass', } );  Once you load that class, you get a config class method which can tell you how that class is configured. However, there’s no guarantee in the Test::DB side that the data is structured the way that I expect. Thus, I need to validate that data and throw an exception immediately if something has changed.  And how do we create our test database?  $dbh-&gt;do(&quot;CREATE DATABASE $db_name&quot;); system(&quot;mysqldump -u $user --password=$password test_db &gt; $filename&quot;) == 0 or croak(&quot;mysqldump failed: $?&quot;); system(&quot;mysql -u $user --password=$password $db_name &lt; $filename&quot;) == 0 or croak(&quot;importing schema to mysql failed: $?&quot;);  The CREATE DATABASE command is fast, so I’m not worried about that. And the test had a single table with very little data, so this was quick. But for Tau Station  , we have a couple of hundred tables and tons of data. This would be slow. For any reasonably mature system, dumping the database each time would be a bad idea. There are also ways you could easily avoid dumping it multiple times, but that hits the next problem: adding that data to your new test database. That would need to be done for each test and that is not something you can trivially speed up.  For a more robust system, I’d probably create a local database service that would simply build a set number of test databases and have them waiting. The test would request the next test database, the service would regist that the database had been taken, and create a new test database in the background while your test runs. The service would also probably clean up old test databases based on whatever policies you think are appropriate.  No Action At A Distance  This line is terrible:  $connect_info-&gt;{dsn} = &quot;dbi:mysql:$db_name&quot;;  The reason that works is because the config data in Client::Model::ClientDB is global and mutable and $connect_info is merely a reference to that data. Instead, if I have a “database service” that tells the code which database it can use, then Test::DB can call that service, and so can Client::Model::ClientDB . Everything relies on a single source of truth instead of hacking global variables and hoping you don’t mess up.  Don’t Drop The Test Database  If there is one thing which I hate about many testing systems, it’s a watching a test horribly fail, but the database is cleaned up (or dropped) and I can’t see the actual data after the test is done. What I often have to do is fire up the debugger and run the code up to the test failure and grab a database handle and try to inspect the data that way. It’s a mess.  Here, we can fix that by simply dropping this line:  END { $dbh-&gt;do(&quot;DROP DATABASE $db_name&quot;) }  At the beginning and end of every test run, we can diag the test database name and if I need to see if there’s an issue in the database, I can still use it. Our database service would have code to drop the database on:   The next day  The next test run  After exceeding a threshold of databases  ... or whatever else you need   In short, keep the valuable data around for debugging.  Rapid Database Development  The database service solution would also have to tie into your database change management strategy. I heavily use sqitch  to manage database changes and I’ve written a lot of code to support odd edge cases. It wouldn’t be hard to write code to let the database service see if it’s up to date with your local version of sqitch . Whatever database change management strategy you use, it needs to be discoverable to properly automate the database service.  Of course, you think, that’s obvious. Yet you’d be shocked how many times I’ve worked with clients whose database change management strategy involves listing a bunch of SQL files and checking their mtime to see which ones need to be applied to your database. Yikes!  Faster Tests  If this is done well, your tests should also be faster. You won’t have the overhead of transactions beyond what your code already has. Plus, you can avoid issues like this:  sub test_basic_combat_attack_behavior ($test,$) { my $ovid = $test-&gt;load_fixture('character.ovid'); my $winston = $test-&gt;load_fixture('character.winston'); my $station = $test-&gt;load_fixture('station.tau-station'); $test-&gt; move_to($station-&gt;port, $ovid, $winston); ok !$ovid-&gt;attack($winston), 'We should not be able to attack someone on the home station.'; ... }  In the above, we’re loading some fixtures. Sometimes those fixtures are very complicated and loading them takes time. For one client, when I would run $test-&gt;load_all_fixtures('connection'); , that would add an extra couple of seconds to every test which needed to do that.  Instead, pre-built test databases can have the test fixtures already loaded. Further, having a pre-populated database helps your code deal with something closer to a real-world problem instead of dealing with an empty database and not catching corner cases that might cause.  Conclusion  By using a database service which merely hands you a temporary test database, you don’t have to worry about leaving the database a mess, managing transactions in tests, or having nasty hacks in your tests to workaround these issues. Most importantly, you’re not changing the behavior of your code. You just use the database like normal. It might be a little bit more work up front to create that database, but it’s worth the effort.  I really do want to get around to creating a proper database tool like this some day. Today is not that day. But I was delighted how even my quick hack, written in just a couple of minutes, made it so much easier to test my code. I should have done this ages ago.  ","title":"Managing a Test Database","url":"\/articles\/managing-a-test-database.html"},{"body":"   Introduction  Common Test Suite Problems  Tests often emit warnings  Tests often fail  There is little evidence of organization  Much of the testing code is duplicated  Testing fixtures are not used (or poorly used)  Code coverage is poorly understood  They take far too long to run  Recommendations  Aggressively search for and remove duplicated tests.  Use code coverage aggressively  Look for code with \"global\" effects  Inline \"hot\" functions.  Recompile your Perl  Preload modules  Parallel tests  Distributed tests  Devel::CoverX::Covered  Test::Class::Moose     This document is about testing applications&mdash;it's not about how to write tests. Application test suites require a different, more disciplined approach than library test suites. I describe common misfeatures experienced in large application test suites and follow with recommendations on best practices. Much of what I describe below is generic and applies to test suites written in any programming language, despite many examples being written in Perl.  Introduction  I often speak with developers who take a new job and they describe a Web site built out of a bunch of separate scripts scattered randomly through directories, lots of duplicated code, poor use of modules, with embedded SQL and printing HTTP headers and HTML directly. The developers shake their head in despair, but grudgingly admit an upside: job security. New features are time-consuming to add, changes are difficult to implement and may have wide-ranging side-effects, and reorganizing the codebase to have a proper separation of concerns, to make it cheaper and safer to hack on, will take lots and lots of time.  A bunch of randomly scattered scripts, no separation of concerns, lots of duplicated code, poor use of modules, SQL embedded directly in them? Does this sound familiar? It's your standard test suite. We're horrified by this in the code, but don't bat an eyelash at the test suite.  Part of this is because much, if not most, of the testing examples we find focus on testing distributions, not applications. If you were to look at the tests for my module DBIx::Class::EasyFixture , you'd see the following tests:  00-load.t basic.t definitions.t groups.t load_multiple_fixtures.t many_to_many.t no_transactions.t one_to_many.t one_to_one.t  These tests were added one by one, as I added new features to DBIx::Class::EasyFixture and each *.t file represents (more or less) a different feature.  For a small distribution, this isn't too bad because it's very easy to keep it all in your head. With only nine files, it's trivial to glance at them, or grep them, to figure out where the relevant tests are. Applications, however, are a different story. This is the number of files from one of my customer's test suites:  $ find t -type f | wc -l 288  That's actually fairly small. One codebase I worked on had close to a million lines of code with thousands of test scripts. You couldn't hold the codebase in your head, you're couldn't glance at the tests to figure out what went where, nor was grepping necessarily going to tell you as tests for particular sections of code were often scattered around multiple test scripts. And, of course, I regularly hear the lament I've heard at many shops with larger codebases: where are the tests for feature X ? Instead of just sitting down and writing code, the developers are hunting for the tests, wondering if there are any tests for the feature they're working on and, if not, trying to figure out where to put their new tests.  Unfortunately, this disorganization is only the start of the problem.  Common Test Suite Problems  I've worked with many companies with large test suites and they tend to share some common problems. I list them in below in the order I try to address these problems (in other words, roughly easiest to hardest).   Tests often emit warnings  Tests often fail (\"oh, that sometimes fails. Ignore it.\")  There is little evidence of organization  Much of the testing code is duplicated  Testing fixtures are not used (or poorly used)  Code coverage is spotty  They take far too long to run   Problems are one thing, but what features do we want to see in large-scale test suites?   Tests should be very easy to write and run  They should run relatively quickly  The order in which tests run should not matter  Test output should be clean  It should be obvious where to find tests for a particular piece of code  Testing code should not be duplicated  Code coverage should be able to analyze different aspects of the system   Let's take a look at some of the problems and try to understand their impacts. While it's good to push a test suite into a desirable state, often this is risky if the underlying problems are ignored. I will offer recommendations for resolving each problem, but it's important to understand that these are recommendations . They may not apply to your situation.  Tests often emit warnings  This seems rather innocuous. Sure, code emits warnings and we're used to that. Unfortunately, we sometimes forget that warnings are warnings : there might very well be something wrong. In my time at the BBC, one of the first things I did was try to clean up all of the warnings. One was a normal warning about use of an undefined variable, but it was unclear to me from the code if this should be an acceptable condition. Another developer looked at it with me and realized that the variable should never be undefined: this warning was masking a very serious bug in the code, but the particular condition was not explicitly tested. By rigorously eliminating all warnings, we found it easier to make our code more correct, and in those places where things were dodgy, comments were inserted into the code to explain why warnings were suppressed. In short: the code became easier to maintain.  Another issue with warnings in the test suite is that they condition developers to ignore warnings. We get so used to them that we stop reading them, even if something serious is going on (on a related note, I often listen to developers complain about stack traces, but a careful reading of a stack trace will often reveal the exact cause of the exception). New warnings crop up, warnings change, but developers conditioned to ignore them often overlook serious issues with their code.  Recommendation : Eliminate all warnings from your test suite, but investigate each one to understand if it reflects a serious issue. Also, some tests will capture STDERR, effectively hiding warnings. Making warnings fatal while running tests can help to overcome this problem.  Tests often fail  For one client, their hour-long test suite had many failing tests. When I first started working on it, I had a developer walk me through all of the failures and explain why they failed and why they were hard to fix. Obviously this is a far more serious problem than warnings, but in the minds of the developers, they were under constant deadline pressures and as far as management was concerned, the test suite was a luxury to keep developers happy, not \"serious code.\" As a result, developers learned to recognize these failures and consoled themselves with the thought that they understood the underlying issues.  Of course, that's not really how it works. The developer explaining the test failures admitted that he didn't understand some of them and with longer test suites that routinely fail, more failures tend to crop up. Developers conditioned to accept failures tend not to notice them. They kick off the test suite, run and grab some coffee and later glance over results to see if they look reasonable (that's assuming they run all of the tests, something which often stops happening at this point). What's worse, continuous integration tools are often built to accomodate this. From the Jenkin's xUnit Plugin page :   Features   Records xUnit tests  Mark the build unstable or fail according to threshold values    In other words, there's an \"acceptable\" level of failure. What's the acceptable level of failure when you debit someone's credit card, or you're sending their medical records to someone, or you're writing embedded software that can't be easily updated?  Dogmatism aside, you can make a case for acceptable levels of test failure, but you need to understand the risks and be prepared to accept them. However, for the purposes of this document, we'll assume that the acceptable level of failure is zero.  If you absolutely cannot fix a particular failure, you should at least mark the test as TODO so that the test suite can pass. Not only does this help to guide you to a clean test suite, the TODO reason is generally embedded in the test, giving the next developer a clue what's going on.  Recommendation : Do not allow any failing tests. If tests fail which do not impact the correctness of the application (such as documentation or \"coding style\" tests), they should be separated from your regular tests in some manner and your systems should recognize that it's OK for them to fail.  There is little evidence of organization  As mentioned previously, a common lament amongst developers is the difficulty of finding tests for the code they're working on. Consider the case of HTML::TokeParser::Simple . The library is organized like this:  lib\/ └── HTML └── TokeParser ├── Simple │ ├── Token │ │ ├── Comment.pm │ │ ├── Declaration.pm │ │ ├── ProcessInstruction.pm │ │ ├── Tag │ │ │ ├── End.pm │ │ │ └── Start.pm │ │ ├── Tag.pm │ │ └── Text.pm │ └── Token.pm └── Simple.pm  There's a class in there named HTML::TokeParser::Simple::Token::ProcessInstruction . Where, in the following tests, would you find the tests for process instructions?  t\/ ├── constructor.t ├── get_tag.t ├── get_token.t ├── internals.t └── munge_html.t  You might think it's in the get_token.t test, but are you sure? And what's that strange munge_html.t test? Or the internals.t test? As mentioned, for a small library, this really isn't too bad. However, what if we reorganized our tests to reflect our library hierarchy?  t\/ └── tests\/ └── html\/ └── tokeparser\/ ├── simple\/ │ ├── token\/ │ │ ├── comment.t │ │ ├── declaration.t │ │ ├── tag\/ │ │ │ ├── end.t │ │ │ └── start.t │ │ ├── tag.t │ │ └── text.t │ └── token.t └── simple.t  It's clear that the tests for HTML::TokeParser::Simple::Token::Tag::Start are in t\/tests\/html\/tokeparser\/simple\/token\/tag\/start.t . And you can see easily that there is no file for processinstruction.t . This test organization not only makes it easy to find where your tests are, it's also easy to program your editor to automatically switch between the code and the tests for the code. For large test suites, this saves a huge amount of time. When I reorganized the test suite of the BBC's central metadata repository, PIPs , I followed a similar pattern and it made our life much easier.  ( Note : the comment about programming your editor is important. Effective use of your editor\/IDE is one of the most powerful tools in a developer's toolbox.)  Of course, your test suite could easily be more complicated and your top-level directories inside of your test directory may be structured differently:  t\/ ├── unit\/ ├── integration\/ ├── api\/ └── web\/  Recommendation : Organize your test files to have a predictable, discoverable structure. The test suite should be much easier to work with.  Much of the testing code is duplicated  We're aghast that that people routinely cut-n-paste their application code, but we don't even notice when people do this in their test code. More than once I've worked on a test suite with a significant logic change and I've had to find this duplicated code and either change it many places or try to refactor it so that it's in a single place and then change it. We already know why duplicated code is bad, I'm unsure why we tolerate this in test suites.  Much of my work in tests has been to reduce this duplication. For example, many test scripts list the same set of modules at the top. I did a heuristic analysis of tests on the CPAN and chose the most popular testing modules and that allowed me to change this:  use strict; use warnings; use Test::Exception; use Test::Differences; use Test::Deep; use Test::Warn; use Test::More tests =&gt; 42;  To this:  use Test::Most tests =&gt; 42;  You can easily use similar strategies to bundle up common testing modules into a single testing module that all of your tests use. Less boilerplate and you can easily dive into testing.  Or as a more egregious example, I often see something like this (a silly example just for illustration purposes):  set_up_some_data($id); my $object = Object-&gt;new($id); is $object-&gt;attr1, $expected1, 'attr1 works'; is $object-&gt;attr2, $expected2, 'attr2 works'; is $object-&gt;attr3, $expected3, 'attr3 works'; is $object-&gt;attr4, $expected4, 'attr4 works'; is $object-&gt;attr5, $expected5, 'attr5 works';  And then a few lines later:  set_up_some_data($new_id); $object = Object-&gt;new($new_id); is $object-&gt;attr1, $new_expected1, 'attr1 works'; is $object-&gt;attr2, $new_expected2, 'attr2 works'; is $object-&gt;attr3, $new_expected3, 'attr3 works'; is $object-&gt;attr4, $new_expected4, 'attr4 works'; is $object-&gt;attr5, $new_expected5, 'attr5 works';  And then a few lines later, the same thing ...  And in another test file, the same thing ...  Put that in its own test function and wrap those attribute tests in a loop. If this pattern is repeated in different test files, put it in a custom test library:  sub test_fetching_by_id ( $class, $id, $tests ) { my $object = $class-&gt;new($id); # this causes diagnostics to display the file and line number of the # caller on failure, rather than reporting *this* file and line number local $Test::Builder::Level = $Test::Builder::Level + 1; foreach my $test (@$tests) { my ( $attribute, $expected ) = @$test; is $object-&gt;$attribute, $expected, \"$attribute works for $class $id\"; } }  And then you call it like this:  my @id_tests = ( { id =&gt; $id, tests =&gt; [ [ attr1 =&gt; $expected1 ], [ attr2 =&gt; $expected2 ], [ attr3 =&gt; $expected3 ], [ attr4 =&gt; $expected4 ], [ attr5 =&gt; $expected5 ], ] }, { id =&gt; $new_id, tests =&gt; [ [ attr1 =&gt; $new_expected1 ], [ attr2 =&gt; $new_expected2 ], [ attr3 =&gt; $new_expected3 ], [ attr4 =&gt; $new_expected4 ], [ attr5 =&gt; $new_expected5 ], ] }, ); for my $test ( @id_tests ){ test_fetching_by_id( 'Object', $test-&gt;{id}, $tests-&gt;{test} ); }  This is a cleanly refactored data-driven approach. By not repeating yourself, if you need to test new attributes, you can just add an extra line to the data structures and the code remains the same. Or, if you need to change the logic, you only have one spot in your code where this is done. Once a developer understands the test_fetching_by_id() function, they can reuse this understanding in multiple places. Further, it makes it easier to find patterns in your code and any competent programmer is always on the lookout for patterns because those are signposts leading to cleaner designs.  Recommendation : Keep your test code as clean as your application code.  Testing fixtures are not used (or poorly used)  One difference between your application code and the test suite is in an application, we often have no idea what the data will be and we try to have a clean separation of data and code.  In your test suite, we also want a clean separation of data and code (in my experience, this is very hit-or-miss), but we often need to know the data we have. We set up data to run tests against to ensure that we can test various conditions. Can we give a customer a birthday discount if they were born February 29th? Can a customer with an overdue library book check out another? If our employee number is no longer in the database, is our code properly deleted, along with the backups and the git history erased? (kidding!)  When we set up the data for these known conditions under which to test, we call the data a test fixture . Test fixtures, when properly designed, allow us generate clean, understandable tests and make it easy to write tests for unusual conditions that may otherwise be hard to analyze.  There are several common anti-patterns I see in fixtures.   Hard to set up and use  Adding them to the database and not rolling them back  Loading all your test data at once with no granularity   In reviewing various fixture modules on the CPAN and for clients I have worked with, much of the above routinely holds true. On top of that, documentation is often rather sparse or non-existent. Here's a (pseudo-code) example of an almost undocumented fixture system for one client I worked with and it exemplified common issues in this area.  load_fixture( database =&gt; 'sales', client =&gt; $client_id, datasets =&gt; [qw\/ customers orders items order_items referrals \/], );  This had several problems, all of which could be easily corrected as code , but they built a test suite around these problems and had backed themselves into a corner, making their test suite dependent on bad behavior.  The business case is that my client had a product serving multiple customers and each customer would have multiple separate databases. In the above, client $client_id connects to their sales database and we load several test datasets and run tests against them. However, loading of data was not done in a transaction, meaning that there was no isolation between different test cases in the same process. More than once I caught issues where running an individual test case would often fail because it depended on data loaded by a different test case, but it wasn't always clear which test cases were coupled with which.  Another issue is that fixtures were not fine-tuned to address particular test cases. Instead, if you loaded \"customers\" or \"referrals\", you got all of them in the database. Do you need a database with a single customer with a single order and only one order item on it to test that obscure bug that occurs when a client first uses your software? There really wasn't any clean way of doing that; data was loaded in an \"all or nothing\" context. Even if you violated the paradigm and tried to create fine-tuned fixtures, it was very hard to write them due to the obscure, undocumented format needed to craft the data files for them.  Because transactions were not used and changes could not be rolled back, each *.t file would rebuild its own test database, a very slow process. Further, due to lack of documentation about the fixtures, it was often difficult to figure out which combination of fixtures to load to test a given feature. Part of this is simply due to the complex nature of the business rules, but the core issues stemmed from a poor understanding of fixtures. This client now has multiple large, slow test suites, spread across multiple repositories, all of which constantly tear down and set up databases and load large amounts of data. The test suites are both slow and fragile The time and expense to fix this problem is considerable due to how long they've pushed forward with this substandard setup.  What you generally want is the ability to easily create understandable fixtures which are loaded in a transaction, tests are run, and then changes are rolled back. The fixtures need to be fine-grained so you can tune them for a particular test case.  One attempt I've made to fix this situation is releasing DBIx::Class::EasyFixture , along with a tutorial . It does rely on DBIx::Class , the most popular ORM for Perl. This will likely make it unsuitable for some use cases.  Using them is very simple:  my $fixtures = DBIx::Class::EasyFixture-&gt;new(schema =&gt; $schema); $fixtures-&gt;load('customer_with_order_without_items'); # run your tests $fixtures-&gt;unload; # also unloads when out of scope  For the customer's code, we could satisfy the different database requirements by passing in different schemas. Other (well-documented) solutions, particularly those which are pure DBI based are welcome in this area.  Recommendation : Fine-grained, well-documented fixtures which are easy to create and easy to clean up.  Code coverage is poorly understood  Consider the following code:  float recip(float number) { return 1.0 \/ number; }  And a sample test:  assert recip(2.0) returns .5;  Congratulations! You now have 100% code coverage of that function.  For a statically typed language, I'm probably going to be moderately comfortable with that test. Alas, for dynamically typed languages we're fooling ourselves. An equivalent function in Perl will pass that test if we use recip(\"2 apples\") as the argument. And what happens if we pass a file handle? And would a Unicode number work? What happens if we pass no arguments? Perl is powerful and lets us write code quickly, but there's a price: it expects us to know what we're doing and passing unexpected kinds of data is a very common source of errors, but one that 100% code coverage will never (no pun intended) uncover. This can lead to false confidence.  To work around false confidence in your code, always assume that you write applications to create things and you write tests to destroy them. Testing is, and should be, an act of violence. If you're not breaking anything with your tests, you're probably doing it wrong.  Or what if you have that code in a huge test suite, but it's dead code? We tend to blindly run code coverage over our entire test suite, never considering whether or not we're testing dead code. This is because we slop our unit, integration, API and other tests all together.  Or consider the following test case:  sub test_forum : Tests(1) ($self) { my $site = $self-&gt;test_website; $site-&gt;login($user, $pass); $site-&gt;get('\/forum'); $site-&gt;follow_link( text =&gt; 'Off Topic' ); $site-&gt;post_ok({ title =&gt; 'What is this?', body =&gt; 'This is a test'. }, 'We should be able to post to the forum'); }  Devel::Cover doesn't know which code is test code and which is not. Devel::Cover merely tells you if your application code was exercised in your tests. You can annotate your code with \"uncoverable\" directives to tell Devel::Cover to ignore the following code, but that potentially means sprinkling your code with annotations all over the place.  There are multiple strategies to deal with this. One of the simplest is to merely run your code coverage tools over the public-facing portions of your code, such as web or API tests. If you find uncovered code, you either have code that is not fully tested (in the sense that you don't know if your API can really use that code) or, if you cannot write an API test to reach that code, investigate if it is dead code.  You can do this by grouping your tests into subdirectories:  t\/ |--api\/ |--integration\/ `--unit\/  Alternatively, if you use Test::Class::Moose , you can tag your tests and only run coverage over tests including the tags you wish to test:  My::Test::Class::Moose-&gt;new({ include_tags =&gt; [qw\/api\/], })-&gt;runtests;  If you start tagging your tests by the subsystems they are testing, you can then start running code coverage on specific subsystems to determine which ones are poorly tested.  Recommendation : Run coverage over public-facing code and on different subsystems to find poor coverage.  They take far too long to run  The problem with long-running test suites is well known, but it's worth covering this again here. These are problems that others have discussed and that I have also personally experienced many times.   With apologies to XKCD  In the best case scenario for developers who always run that long-running test suite, expensive developer time is wasted while the test suite is running. When they launch that hour-long (or more) test suite, they frequently take a break, talk to (read: interrupt) other developers, check their Facebook, or do any number of things which equate to \"not writing software.\" Yes, some of those things involve meetings or research, but meetings don't conveniently schedule themselves when we run tests and for mature products (those which are more likely to have long-running test suites), there's often not that much research we really need to do.  Here are some of the issues with long-running test suites:   Expensive developer time is wasted while the test suite runs  Developers often don't run the entire test suite  Expensive code coverage is not generated as a result  Code is fragile as a result   What I find particularly curious is that we accept this state of affairs. Even a back-of-the-envelope calculation can quickly show significant productivity benefits that will pay off in the long run by taking care of our test suite. I once reduced a BBC test suite's run time from one hour and twenty minutes down to twelve minutes ( Note: today I use a saner approach that results in similar or greater performance benefits ). We had six developers on that team. When the test suite took over an hour to run, they often didn't run the test suite. They would run tests on their section of code and push their code when they were comfortable with it. This led to other developers finding buggy code and wasting time trying to figure out how they broken it when, in fact, someone else broke the code.  But let's assume each developer was running the test suite at least once a day (I'm careful about testing and often ran mine twice a day). By cutting test suite run time by over an hour, we reclaimed a full day of developer productivity every day! Even if it takes a developer a month to increase perfomance by that amount it pays for itself many times over very quickly. Why would you not do this? As a business owner, wouldn't you want your developers to save time on their test suite so they can create features faster for you?  There are several reasons why this is difficult. Tasking a developer with a block of time to speed up a test suite means the developer is not creating user-visible features during that time. For larger test suites, it's often impossible to know in advance just how much time you can save or how long it will take you to reach your goal. In most companies I've worked with, the people who can make the decision to speed up the test suite are often not the people feeling the pain. Productivity and quality decrease slowly over time, leading to the boiling frog problem .  What's worse: in order to speed up your test suite without affecting behavior, the test suite often has to be \"fixed\" (eliminating warnings, failures, and reducing duplication) to ensure that no behavior has been changed during the refactor.  Finally, some developers simply don't have the background necessary to implement performance optimizations. While performance profiles such as Perl's Devel::NYTProf can easily point out problem areas in the code, it's not always clear how to overcome the discovered limitations.  The single biggest factor in poor test suite performance for applications is frequently I\/O. In particular, working with the database tends to be a bottleneck and there's only so much database tuning that can be done. After you've profiled your SQL and optimized it, several database-related optimizations which can be considered are:   Using transactions to clean up your database rather than rebuilding the database  Only connect to the database once per test suite (hard when you're using a separate process per test file)  If you must rebuild the database, maintain a pool of test databases and assign them as needed, rebuilding used ones in the background  Use smaller database fixtures instead of loading everything at once   After you've done all you can to improve your database access, you may find that your test suite is \"fast enough\", but if you wish to go further, there are several steps you can take.  Recommendations  Aggressively search for and remove duplicated tests.  For poorly organized test suites, developers sometimes make the mistake of putting tests for something in a new *.t file or add them to a different *.t file, even if related tests already exist. This strategy can be time-consuming and often does not result in quick wins.  Use code coverage aggressively  For one test suite, I found that we were using a pure Perl implementation of JSON. As the test suite used JSON extensively, switching to JSON::XS gave us a nice performance boost. We may not have noticed that if we hadn't been profiling our code.  Look for code with \"global\" effects  On one test suite, I ensured that Universal::isa and Universal::can cannot be loaded. It was a quick fix and sped up the test suite by 2% (several small accumulations of improvements can add up quickly).  Inline \"hot\" functions.  Consider the following code which runs in about 3.2 seconds on my computer:  #!\/usr\/bin\/env perl use strict; use warnings; no warnings 'recursion'; for my $i ( 1 .. 40 ) { for my $j ( 1 .. $i**2 ) { my $y = factorial($j); } } sub factorial { my $num = shift; return 1 if $num &lt;= 1; return $num * factorial($num - 1); }  By rewriting the recursive function as a loop, the code takes about .87 seconds:  sub factorial { my $num = shift; return 1 if $num &lt;= 1; $num *= $_ for 2 .. $num - 1; return $num; }  By inlining the calculation, the code completes in .69 seconds:  for my $i ( 1 .. 40 ) { for my $j ( 1 .. $i**2 ) { my $y = $j; if ( $y &gt; 1 ) { $y *= $_ for 2 .. $y - 1; } } }  In other words, in our trivial example, the inlined behavior is roughly 20% faster than the iterative function and 80% faster than the recursive function.  Recompile your Perl  You may wish to recompile your Perl to gain a performance improvement. Many Linux distributions ship with a threaded Perl by default. Depending on the version of Perl you ship with, you can gain performance improvements of up to 30% by recompiling without threads. Of course, if you use threads, you'll feel very stupid for doing this. However, if you don't make heavy use of threads, switching to a forking model for the threaded code may make the recompile worth it. Naturally, you'll need to heavily benchmark your code (preferably under production-like loads) to understand the trade-offs here.  Preload modules  If your codebase makes heavy use of modules that are slow to load, such as Moose , Catalyst , DBIx::Class and others, preloading them might help. forkprove is a utility written by Tatsuhiko Miyagawa that allows you to preload slow-loading modules and then forks off multiple processes to run your tests. Using this tool, I reduced one sample test suite's run time from 12 minutes to about a minute . Unfortunately, forkprove doesn't allow schedules, a key component often needed for larger test suites. I'll explain that in the next section.  Parallel tests  Running tests in parallel is tricky. Some tests simply can't be run with other tests. Usually these are tests which alter global state in some manner that other processes will pick up, or might cause resource starvation of some kind.  Or some tests can be run in parallel with other tests, but if several tests are updating the same records in the database at the same time, locking behavior might slow down the tests considerably.  Or maybe you're running 4 jobs, but all of your slowest tests are grouped in the same job: not good.  To deal with this, you can create a schedule that assigns different tests to different jobs, based on a set of criteria, and then puts tests which cannot run in parallel in a single job that runs after the others have completed.  You can use TAP::Parser::Scheduler to create an effective parallel testing setup. You can use this with TAP::Parser::Multiplexer to create your parallel tests. Unfortunately, as of this writing there's a bug in the Multiplexer whereby it uses select in a loop to read the parser output. If one parser blocks, none of the other output is read. Further, the schedule must be created prior to loading your test code, meaning that if your tests would prefer a different schedule, you're out of luck. Also, make test currently doesn't handle this well. There is work being done by David Golden to alleviate this problem.  My preferred solution is to use Test::Class::Moose . That has built-in parallel testing and writing schedules is very easy. Further, different test cases can simply use a Tags(noparallel) attribute to ensure that they're run sequentially after the parallel tests.  Aside from the regular benefits of Test::Class::Moose , an interesting benefit of this module is that it loads all of your test and application code into a single process and then forks off subprocesses. As a result, your code is loaded once and only once. Alternate strategies which try to fork before loading your code might still cause the code to be loaded multiple times.  I have used this strategy to reduce a 12 minute test suite to 30 seconds .  Distributed tests  Though I haven't used this module, Alex Vandiver has written TAP::Harness::Remote . This module allows you to rsync directory trees to multiple servers and run tests on those servers. Obviously, this requires multiple servers.  If you want to roll your own version of this, I've also released TAP::Stream , a module that allows you to take streams (the text, actually) of TAP from multiple sources and combine them into a single TAP document.  Devel::CoverX::Covered  There is yet another interesting strategy: only run tests that exercise the code that you're changing. Johan Lindström wrote Devel::CoverX::Covered . This modules is used in conjunction with Paul Johnson's Devel::Cover to identify all the places in your tests which cover a particular piece of code. In the past, I've written tools for vim to read this data and only run relevant tests. This is a generally useful approach, but there are a couple of pitfalls.  First, if you test suite takes a long time to run, it will take much, much longer to run with Devel::Cover . As a result, I recommend that this be used with a special nightly \"cover build\" and have the results synched back to the developers.  Second, when changing code, it's easy to change which tests cover your code, leading to times when this technique won't cover your actual changes thoroughly. In practice, this hasn't been a problem for me, but I've not used it enough to say that with confidence.  Recommendation : Don't settle for slow test suites. Pick a goal and work to achieving that goal (it's easy to keep optimizing for too long and start getting diminishing marginal returns).  Test::Class::Moose  If you start creating a large Web site, do you start writing a bunch of individual scripts, each designed to handle one URL and each handling their own database access and printing their output directly to STDOUT? Of course not. Today, professional developers reach for Sinatra, Seaside, Catalyst, Ruby on Rails or other Web frameworks. They take a bit more time to set up and configure, but we know they generally save more time in the long run. Why wouldn't you do that with your test suite?  If you're using Perl, many of the problems listed in this document can be avoided by switching to Test::Class::Moose . This is a testing framework I designed to make it very easy to test applications. Once you understand it, it's actually easy to use for testing libraries, but it really shines for application testing.  Note that I now regret putting Moose in the name. Test::Class::Moose is a rewrite of Test::Class using Moose , but it's not limited to testing Moose applications. It uses Moose because internally it relies on the Moose meta-object protocol for introspection.  Out of the box you get:   Reporting  Parallel tests (which optionally accepts a custom schedule)  Tagging tests (slice and dice your test suite!)  Test inheritance (xUnit for the win!)  Full Moose support  Test control methods (startup, setup, teardown, shutdown)  Extensibility  All the testing functions and behavior from Test::Most   To learn about xUnit testing in Perl, you may wish to read a five-part tutorial I published at Modern Perl Books:   Organizing test suites with Test::Class  Reusing test code  Making your testing life easier  Using test control methods  Working with Test::Class test suites   That tutorial is slightly out of date (I wrote it a few years ago), but it explains effective use of Test::Class and some common anti-patterns when using it.  Doug Bell has started a tutorial for Test::Class::Moose . That also needs updating, but between those and reading the Test::Class::Moose documentation, you should be able to get up to speed fairly quickly.  ","title":"The Zen of Test Suites","url":"\/articles\/zen-of-test-suites.html"},{"body":"    Consistency Is Your Friend  Language Design  Who the Hell is Kim?  This Ain’t Kim  Turning Corinna into Kim  Conclusion     Consistency Is Your Friend  Note : None of the following is set in stone. This is merely a way to start discussion on some issues with syntactic inconsistencies in Corinna.  Years ago I was working with some graphing software where points would be declared via:  var point = point(2,3);  Except that the point function took Y,X as arguments, not the near universally-accepted X,Y . To this day, I loathe the author of that library for making my life hell.  On a more personal note, while building Tau Station  —a narrative sci-fi MMORPG—I was trying to respect proper separation of concerns and not building god objects, so when it came time to check if an NPC has a mission for a character, I had this:  if ( $missions-&gt;has_mission_for( $character, $npc ) ) { ... }  Or was it this?  if ( $missions-&gt;has_mission_for( $npc, $character ) ) { ... }  To this day, I cannot remember which style I used and what justification I had for that, but it was a constant source of bugs because I kept getting the $npc and $character swapped. So I bit the bullet and decided that, as much as possible, Tau Station code would follow SVO (subject-verb-object) syntax, instead of the weird “context-verb-subject-object” I had created. Or was it “context-verb-object-subject”? Who knows? Instead, the above became this:  if ( $npc-&gt;has_missions_for($character) ) { ... }  And now the meaning of the code is crystal clear, though people will complain about “God objects” ( which I address here  ).  Consistency brings clarity. Reasoning about code is hard enough when it appears to be an ad-hoc mishmash of rules.    So, um, that brings us to the Corinna project  , my attempt to bring modern object-oriented programming to the Perl language.  Language Design  As I’ve previously pointed out, my first foray into language design didn’t go as well as I hoped. At its heart, you can think of language design as consisting of three parts:   Semantics  Syntax  Community   I mostly had the semantics and syntax down, but I hadn’t done a great job of paying attention to the community. The community feedback was mixed , to put it politely. So an IRC channel was created (irc.perl.org #cor) and a github project  was started and anyone with an interest in the project was welcome to share their thoughts.  By including the community, we’ve done a fantastic job of turning Corinna from an interesting sideshow to something that is (mostly) popular with the Perl community. If we can bring her over the finish line, we have something that is likely to be included in the Perl core as an experimental project. You have to ask for it by name:  use feature 'class'; class Foo { ... }  And then you get all the lovely OO goodies in a powerful, declarative syntax.  We’re soooooo close to being able to propose an RFC that I am loathe to throw a wrench in the works, but I must. I must because Kim is very, very unhappy with Corinna.  Who the Hell is Kim?  Please note that much of this is largely derived from emails I’ve swapped with Damian Conway  . I’ve rewritten it considerably, but the core of this is him noting inconsistencies that have been discussed on the IRC channel, but not put together conherently. Corinna has evolved with some serious inconsistencies and the time to consider fixing them is now, not after it’s released.  Corinna, as some of you may know, was chosen as the name of this project because my nom de guerre ,   is “Ovid” and he wrote poems to Corinna, a love interest (if you’re familiar with the poetry, you know I’m keeping this family-friendly). Kim isn’t jealous of Corinna. She’s furious because Corinna, after literally years of design, is breaking some cardinal rules of consistency. So, let’s dig into that.  So let’s look at the way we declare things in Perl. In particular, we’ll look at my variables, subroutines, and packages.  my $answer = 42; sub douglas_adams { ... } package Restaurant { ... }  Those all seem different, but we start declaring what type of thing we have: a variable, a subroutine, or a package. Then we follow that up with the name of the thing, followed by the optional set up of that thing.  But what if we want to modify how those things behave? Attributes have been alternately proposed and rejected by many developers, but if you use threads::shared in Perl, you can declare a variable as shared across threads:  my $answer :shared = 42;  What if you want to declare a subroutine as an lvalue?  sub douglas_adams :lvalue {...}  Or declaratively state the version of a package without using the procedural $VERSION assignment?  package Restaurant v3.1.4 { ... }  In fact, if we continue down this road, we see a pattern start to emerge:         keyword  identifier  modifiers?  setup?      my  $lexvar  :shared      our  $packvar  :Tracked  = 0    state  $statevar  :Readonly  = 1    sub  action  :lvalue  () {...}    package  Namespace  v1.2.3  {...}    format  Report    = ... .     KIM stands for “Keyword”, “Identifier”, and “Modifier” (yes, it should be “KIMS”, but I like the “Kim versus Corinna” description. Sue me.)  Kim likes consistency. You must always declare the kind of thing you’re going to use and then name it. You can then optionally modify its base behavior and then optionally set it up. Very, very consistent:  KEYWORD IDENTIFIER MODIFIERS SETUP  Of course, you can point to plenty of areas where Perl is not consistent, but Corinna is designed to improve the Perl language, not add to the inconsistency, so let’s look at a few examples:   KIM features in Corinna        keyword  identifier  modifiers?  setup?      role  Tracked    {...}      class  Root    {...}      slot  $msg  :param      slot  $handler  :handles(exists delete)  = Handler->new;    slot  $size  :reader :param  = 100;    slot  $created  :reader  = time;      method  is_root ()    {...}    method  obj_count  :common ()  {...}    method  insert  :private ($key,$value)  {...}    method  show ($msg)    {...}    method  obj_count  :overrides :common ()  {...}     So that’s actually looking pretty good. Corinna is developing with a clear, consistent KIM syntax. But if that was all, this wouldn’t be a helpful article. That’s the good; it’s time to look at the bad. And maybe the ugly.  This Ain’t Kim  Let’s look at a few examples of inconsistencies. We’ll start with putting the modifier first.   Modifiers First        modifier  keyword  identifier  setup      abstract  class  Counter  {...}     And let’s mix up keywords and modifiers.   Keywords as Modifiers        keyword  identifier  keyword as modifiers  setup      class  Handler  isa Counter does Tracked  {...}     Let’s use modifers plus keywords plus identifiers plus modifiers plus ...   Keywords gonna Keyword         modifier  keyword  identifier  modifiers  setup      before  method  obj_count  :overrides :common ()  { warn “Counting...“; }    after  method  obj_count  :overrides :common ()  { warn “...done”; }    around  method  obj_count  :overrides :common ()  { return 1 + $self->$ORIG(); }     So, yeah. Damian pointed all of this out to me and he’s right. Corinna is close , but there are syntactical inconsistencies which, if agreed that they should be addressed, should be addressed before an RFC, not after.  Turning Corinna into Kim  The name’s still Corinna, but she needs KIM.  When I started converting Tau Station code to (in)consistently use SVO syntax, it made it much easier to maintain. Doing the same thing with Corinna is worthwhile.  So, how do we address all these inconsistencies? Corinna semantics are OK. We just need some tweaks to the syntax.         keyword  identifier  modifiers?  setup?      class  Counter  :abstract  {...}    class  Handler  :isa(Counter) :does(Tracked)  {...}    method  obj_count  :before :overrides :common  () { warn “Counting...“; }    method  obj_count  :after :overrides :common  () { warn “...done”; }    method  obj_count  :around :overrides :common  () { return 1 + $self->$ORIG(); }     And here’s a slightly modified example from Damian showing this more consistent syntax:  role Tracked { slot $msg :param; method report () { $self-&gt;show($msg++) } method show; # required } class Root { method is_root () { return 1; } } class Counter :abstract { slot $obj_count :common :reader = 0; ADJUST { $obj_count++ } DESTRUCT { $obj_count-- } } class Handler :isa(Counter) :does(Tracked) { slot $handler :handles(exists delete) = Handler-&gt;new; slot $size :reader :param = 100; slot $created :reader = time; ADJUST { croak(&quot;Too small&quot;) if $size &lt; 1; } DESTRUCT { $handler-&gt;shutdown; } method insert :private ($key, $value ) { if ( ! $self-&gt;exists($key) ) { $handler-&gt;set( $key, $value ); } } method show ($msg) { say $msg; } method obj_count :common :overrides () { $self-&gt;next::method() - 42; } method obj_count :common :before () { warn &quot;Counting...&quot;; } method obj_count :common :around () { return 1 + $self-&gt;$ORIG(); } method obj_count :common :after () { warn &quot;...done&quot;; } }  Conclusion  As you can see from the sample code, this still looks like Corinna. We have the semantics and syntax, but do we have the community? The changes aren’t huge, but they might prove controversial because Corinna would again be changing. But I think it’s changing for the better. Further, by establishing the KIM principle now, further upgrades to the Perl language can have guidance on maintaining consistency.  ","title":"Language Design Consistency","url":"\/articles\/language-design-consistency.html"},{"body":" If you search LinkedIn for groups related to Scrum, you get almost 700 results as of this writing. Many of these are fantastic groups, with great people offering advice from the trenches. If you use Agile, particularly Scrum, you should follow these groups. Just beware that there is a problem there: sometimes you find yourself staring into the wide eyes of the converted. It's particularly bad if you're staring in a mirror.  From time to time people in various Scrum groups ask \"What are the weakness of Scrum?\" And, I kid you not, more than once I have seen the reply \"none, Scrum is perfect.\" Those respondents are the people who won't change their mind and won't change the subject and I very much want to gently shove them into a meeting full of their fellow converted and go back to getting work done. Out of deference to their disability, I will set a goal for the meeting, time-box it at eight hours, and when the participants email me the action items, I will print out that email and burn it. It will be a good day.  Or, as a fellow Agile trainer wrote to me \"some of my clients are dumping Scrum just to get away from the zealots.\"  Lord, save me from your followers.  Fortunately, those who believe that Scrum (or XP, or Kanban, or Scrumban, or, or, or ...) is perfect, are in the minority. They're the obvious crackpots that we greet with a fixed smile and back away from slowly, but there are the less obvious crackpots who are far more dangerous because they sound so reasonable.  These are the ones who believe with a fiery passion that everything should be Agile. These people are going to hurt your company.  To understand this, we need to remember what Agile is. Agile development is simply a set of practices designed to better cope with uncertainty and change. That's all. A key indicator of when to use Agile is when you're developing a new technology and saying \"wow, look at all the possibilities!\" When you're breaking new ground, when you're pushing change rather than following the herd, then Agile is a great approach.  But what if you don't have a rapidly changing environment? What if quality and reproducibility are more important considerations? Janitors don't need product owners. They don't create \"who\/what\/why\" story cards. They don't have sprints. If your janitorial staff are struggling with a chaotic, uncertain environment, you have bigger problems on your hands. Mind you, there are janitors who have to deal with rapidly changing environments and uncertainty. We call them SEAL teams.  OK, so maybe you've outsourced your janitorial staff and are thinking everything else can be Agile. Let's head over to your accounting department. Forget about your prejudices for a moment and look at what accounting does. Ask yourself one question \"do I want them using a project management system that's optimized for uncertainty and change, or quality and reproducibility?\" Remember, your answer not only has financial implications, but legal ones as well.  When it's phrased in those terms, the answer is fairly clear. Even the most ardent Agile proponents would have to agree that an accounting team may need to be run differently from a software development team. There is no \"one size fits all\" suit for project management and if you're trying to cram everything in your organization into the same system, you're begging for trouble.  As a rule of thumb, for every department ask yourself how \"new\" their product is. The newer it is, the more likely it's being born in an uncertain environment and Agile is a good approach. If it's an older, mature, stable environment, consider PRINCE2, TQM, or other related project management solutions. Lean solutions fit somewhere in the middle of those two extremes.  Don't let yourself get caught up in the hype. Agile solutions are excellent, but don't go out in pairs, knocking on doors and preaching to the masses. Different problems require different solutions and we need to give up our ever present quest to find the One True Way.  If you'd like to know more, you might want to read when to choose agile  ","title":"When Going Agile Can Hurt Your Company","url":"\/articles\/going-agile-can-hurt-your-company.html"},{"body":" Yeah, sit back in your chair, you silly, naïve developer. I've been building software since before you were potty-trained and I know what I'm doing. I know what customers want and I know how to build it for them.  Er, except for the fact that I'm usually wrong.  When I first encountered A\/B testing, I had fixed a bug in a search engine and tried to push my change. It was rejected. Why? Because I hadn't wrapped it in an A\/B test. I was confused. Why would I want to A\/B test a bugfix? Under certain conditions our search engine would return no results and I fixed it to ensure that it would not only return results, but they would be relevant results.  Boss: I don't care. A\/B test it anyway.  So I did. And it was painful, waiting days for the results to come in ... and being dismayed to see very clearly that improving the behavior of the search engine led to significantly lower customer conversion. I even questioned the validity of the data, but to no avail. It took me a long time to truly realize the implications, but it harmonizes quite well with other things I teach about software:  Software behavior should be considered buggy when it causes unwanted customer behavior, not unwanted software behavior.  Yes, that's an oversimplification, but bear with me.  A\/B Testing: Slayer of Egos  I had a client who introduced a horizontal scroll bar to their e-commerce site and significantly improved sales. Another company found that removing social media buttons increased conversions . And I found that our customers didn't respond well to better search results.  For many people those ideas might sound counter-intuitive, but they had real data to back them up. Instead, here's what often happens when you have experts guiding development by their \"experience\":   We need multiple pictures of our product on the \"Buy now\" page! Sales drop.  We need to show related items so we can cross-sell! Sales drop.  We need to show more relevant search results! Sales drop.   Quite often you won't find out that sales dropped as a result of a particular change because you haven't measured your customer's behavior and ignoring your customers is, by anyone's reckoning, a recipe for disaster. Remember how Digg went from king of the web to court jester in 2010 because they ignored their customers?  A\/B testing is not a silver bullet, but it's an excellent way of getting real, honest, data-driven answers to your questions. And if you have an ego, it's a punch in the gut. Again and again and again. Why? Because the vast majority of A\/B tests are inconclusive (no apparent change in behavior) or negative (you probably lost money). Your customers don't care about how experienced you are.  The Customer Isn't Always Right  Your customers care about what your customers care about. Unfortunately, they often don't know what they care about until they experience it first-hand or some bandwagon effect kicks in and everyone piles on (this can be either good or bad for you).  To give an example of how problematic this can be, consider the case of a company in France a friend of mind was running (sorry, I can't disclose the name). It was was a very innovative paid search engine covering a very technical field. The company raised a lot of seed funding and was growing steadily, but it wasn't profitable because potential customers weren't signing up fast enough. Surveys revealed they wanted the search engine to return research results in French and English. This was because the top research in this field was published in English and the French professionals wanted access to it.  Given that the search engine was in French and it was searching highly technical documents, converting it to also search English documents, search them correctly , buy subscriptions to the various technical sources those documents were found in, and presenting them seamlessly on a French-language website turned out to be both expensive and time-consuming. And it wasn't just development costs: it was ongoing costs to subscribe to, parse, and index the English language materials.  The feature was released to much fanfare and potential customers didn't care. At all. They asked for this feature but it had no significant impact on conversion. As it turns out, French professionals may very well speak English, but they still preferred French. What they wanted wasn't English-language documents; they wanted a reassurance they were getting a good value for their money . Given that the company had a simple metric&mdash;new customer subscriptions&mdash;A\/B testing could have been a far less expensive way of divining this information. Maybe only provide a single English-language source. Maybe provide testimonials. Maybe something as simple as playing with language or images may have had an impact?  Instead, the cost of development, the opportunity cost of not developing other features or spending on marketing, and the ongoing cost of maintaining the English-language corpus were all significant contributing factors to the collapse of the company.  The literature is rife with stories of companies listening to what their customers say instead of paying attention to what their customers do. A\/B testing tells you what they're actually doing.  Limitations of A\/B Testing  A\/B testing is a powerful tool in your toolbox, but it shouldn't be considered the only tool. Further, just as you don't use a hammer with screws, you shouldn't misuse A\/B testing. Evan Miller has a great article entitled How Not To Run an A\/B Test .  But statistical flaws aside, there are other issues with A\/B testing. As part of your agile toolkit, you want to run them often . If you only release new versions of your software quarterly you're going to struggle to rapidly respond to customer needs. If you're testing at the bottom of a conversion funnel and only get 10 visitors a week, you might wait months to get a meaningful result. Or maybe you've run a very successful A\/B test, but it's not related to KPIs of the company.  And the killer, one I've seen all too often: \"that idea is rubbish! We don't need to test that!\"  If you've made A\/B tests painful to set up and can only run them periodically, I can understand that attitude (but the A\/B test isn't the real problem there). However, if you can easily run tests and respond to results quickly, it often makes sense to test \"dumb\" ideas. Consider the case of the horizontal scroll bar I mentioned earlier.  You'll have plenty of experts telling you why horizontal scroll bars are a disaster , so why did it work for my aforementioned client?  First, the key thing to remember is that A\/B tests tell you what your customers are doing, but not why.  With the horizontal scroll bar, the test clearly showed increased conversion, but the designer was extremely unhappy. After a lot of thought and examining the page, she noticed something interesting. A side effect of the horizontal scroll bar was that the full product description was now visible without vertical scrolling. She redesigned the page to use a vertical scroll bar instead of a horizontal one, but kept the full product description visible.  Once again there was another nice increase in conversion rates, significantly better than the original version.  Your expertise isn't in dictating features, it's in designing experiments and interpreting the results.  So you see? Your experience still matters and you can keep feeding your ego, but now you have hard data to back it up.  Summary  If you're not doing A\/B testing, you really should consider it. There are plenty of companies which provide simple tools for integrating A\/B testing into your Web site. Once you understand them, and you get a feel for the power of A\/B testing, it's time for you to start building an internal tool that's more suitable for your needs. You'll be able to test things you could never test before, and you'll have better access to customer segmentation data.  Stop doing ego-driven development. This is a typical story card we see today:  As a &lt;developer&gt; I want to &lt;build random stuff&gt; So that &lt;the boss stays happy&gt;  This is what you're really looking for:  We suspect that &lt;building this feature&gt; ... &lt;for these people&gt; ... will achieve &lt;this measurable result&gt; We will know we succeeded when we see &lt;this market signal&gt;.  ","title":"The Surprises of A\/B Testing","url":"\/articles\/the-surprises-of-ab-testing.html"},{"body":"  Immutable Objects  I’ve been spending time designing Corinna  , a new object system to be shipped with the Perl language. Amongst its many features, it’s designed to make it easier to create immutable objects, but not everyone is happy with that. For example, consider the following class:  class Box { has ($height, $width, $depth) :reader :new; has $volume :reader = $width * $height * $depth; } my $original_box = Box-&gt;new(height=&gt;1, width=&gt;2, depth=&gt;3); my $updated_box = $original_box-&gt;clone(depth=&gt;9); # h=1, w=2, d=9  Because none of the slots have a :writer attribute, there is no way to mutate this object. Instead you call a clone method, supplying an overriding value for the constructor argument you need to change. The $volume argument doesn’t get copied over because it’s derived from the constructor arguments.  But not everyone is happy with this approach  . Aside from arguments about utility of the clone method, the notion that objects should be immutable by default has frustrated some developers reading the Corinna proposal. Even when I point out just adding a :writer attribute is all you need to do to get your mutability, people still object. So let’s have a brief discussion about immutability and why it’s useful.  But first, here’s my last 2020 Perl Conference presentation on Corinna.     The Problem  Imagine, for example, that you have a very simple Customer object:  my $customer = Customer-&gt;new( name =&gt; &quot;Ovid&quot;, birthdate =&gt; DateTime-&gt;new( ... ), );  In the code above, we’ll assume the $customer can give us useful information about the state of that object. For example, we have a section of code guarded by a check to see if they are old enough to drink alcohol:  if ( $ovid-&gt;old_enough_to_drink_alcohol ) { ... }  The above looks innocent enough and it’s the sort of thing we regularly see in code. But then this happens:  if ( $ovid-&gt;old_enough_to_drink_alcohol ) { my $date = $ovid-&gt;birthdate; ... # deep in the bowels of your code my $cutoff_date = $date-&gt;set( year =&gt; $last_year ); # oops! ... }  We had a guard to ensure that this code would not be executed if the customer wasn’t old enough to drink, but now in the middle of that code, due to how DateTime is designed, someone’s set the customer birth date to last year! The code, at this point, is probably in an invalid state and its behavior can no longer be considered correct.  But clearly no one would do something so silly, would they?  Global State  We’ve known about the dangers of global state  for a long time. For example, if I call the following subroutine, will the program halt or not?  sub next ($number) { if ( $ENV{BLESS_ME_LARRY_FOR_I_HAVE_SINNED} ) { die &quot;This was a bad idea.&quot;; } return $number++; }  You literally cannot inspect the above code and tell me if it will die when called because you cannot know, by inspection, what the BLESS_ME_LARRY_FOR_I_HAVE_SINNED environment variable is set to. This is one of the reasons why global environment variables are discouraged.  But here we’re talking about mutable state. You don’t want the above code to die, so you do this:  $ENV{BLESS_ME_LARRY_FOR_I_HAVE_SINNED} = 0; say next(4);  Except that now you’ve altered that mutable state and anything else which relies on that environment variable being set is unpredicatable. So we need to use local to safely change that in the local scope:  { local $ENV{BLESS_ME_LARRY_FOR_I_HAVE_SINNED} = 0; say next(4); }  Even that is not good because there’s no indication of why we’re doing this   , but at least you can see how we can safely change that global variable in our local scope.  ORMs  And I can hear your objection now:   “But Ovid, the DateTime object in your first example isn’t global!”   That’s true. What we had was this:  if ( $ovid-&gt;old_enough_to_drink_alcohol ) { my $date = $ovid-&gt;birthdate; ... # deep in the bowels of your code my $cutoff_date = $date-&gt;set( year =&gt; $last_year ); # oops! ... }  But the offending line should have been this:  # note the clone(). my $cutoff_date = $date-&gt;clone-&gt;set( year =&gt; $last_year );  This is because the set method mutates the object in place, causing everything holding a reference to that object to silently change . It’s not global in the normal sense, but this action at a distance is a source of very real bugs  .  It’s a serious enough problem that DateTime::Moonpig  and DateTimeX::Immutable  have both been written to provide immutable DateTime objects, and that brings me to DBIx::Class  , an excellent ORM for Perl.  As of this writing, it’s been around for about 15 years and provides a component called DBIx::Class::InflateColumn::DateTime  . This allows you to do things like this:  package Event; use base 'DBIx::Class::Core'; __PACKAGE__-&gt;load_components(qw\/InflateColumn::DateTime\/); __PACKAGE__-&gt;add_columns( starts_when =&gt; { data_type =&gt; 'datetime' } create_date =&gt; { data_type =&gt; 'date' } );  Now, whenever you call starts_when or create_date on an Event instance, you’ll get a DateTime object instead of just the raw string from the database. Further, you can set a DateTime object and not worry about your particular database’s date syntax. It just works .  Except that the object is mutable and we don’t want that.   You can fix this by writing your own DBIx::Class component to use immutable DateTime objects.  package My::Schema::Component::ImmutableDateTime; use DateTimeX::Immutable; use parent 'DBIx::Class::InflateColumn::DateTime'; sub _post_inflate_datetime { my ( $self, @args ) = @_; my $dt = $self-&gt;next::method(@args); return DateTimeX::Immutable-&gt;from_object( object =&gt; $dt ); } 1;  And then load this component:  __PACKAGE__-&gt;load_components( qw\/+My::Schema::Component::ImmutableDateTime\/ );  And now, when you fetch your objects from the database, you get nice, immutable DateTime s. And it will be interesting to see where your codebase fails!  Does all of this mean we should never use mutable objects? Of course not. Imagine creating an immutable cache where, if you wanted to add or delete an entry, you had to clone the entire cache to set the new state. That would likely defeat the main purpose of a cache: speeding things up. But in general, immutability is a good thing and is something to strive for. Trying to debug why code far, far away from your code has reset your data is not fun.  ","title":"Why Do We Want Immutable Objects?","url":"\/articles\/using-immutable-datetime-objects-with-dbixclass.html"},{"body":"    Introduction  CamelCaseNames or underscore_names?  Plural or Singular Tables?  Don’t name the ID column “id”  Column Naming  Avoid NULL Values  Database Types  What does a NULL value mean?  NULLs lead to logical impossibilities  Summary     Introduction  When moving from project to project, it’s unfortunate that we find that there are no consistent standards on database design, despite SQL having been around for decades. I suspect that much of this is because most developers don’t understand database design . In fact, with my years of hiring developers, only a handful of times have I met developers who can properly normalize a database. To be fair, normalization can be hard, but most developers I’ve interviewed, even excellent ones with strong SQL skills, don’t have database design skills.  But this article isn’t about database normalization. If you want to learn more, here’s a short talk I gave which explains the basics.     Instead, when you have a working database, the question we want to know is “what standards can we apply which make it easier to use that database?” If these standards were to be widely adopted, databases would be easier to use because you wouldn’t have to learn and remember a new set of standards every time you work with a new database.  CamelCaseNames or underscore_names?  Let’s get this out of the way quickly. I routinely see database examples online where we see table names like CustomerOrders or customer_orders . Which should you use? You probably want to use whatever standard is already in place, but if you’re creating a new database, I recommend using_undercores for accessibility. The words “under value” have a different meaning from the word “undervalue”, but the former, with underscores, is always under_value , while the latter is undervalue . With CamelCase, it’s Undervalue versus UnderValue which, since SQL is case-insensitive, are identical. Further, if you have vision problems and are constantly playing around with fonts and sizes to distinguish words, the underscores are much easier to read.  As a side problem, CamelCase is anecdotally harder to read for people for whom English isn’t their first language.  That being said, this is a personal preference and not a strong recommendation.  Plural or Singular Tables?  There’s long been a huge debate amongst experts in database theory about whether or not database tables should be singular ( customer ) or plural ( customers ). Without going into the theory, let me cut the Gordian Knot with a healthy dose of pragmatism: plural table names are less likely to conflict with reserved keywords.  Do you have users? SQL has a user reserved word. Do you want a table of constraints? constraint is a reserved word. Is audit a reserved word but you want an audit table? By simply using the plural form of nouns, most reserved words won’t cause you grief when you’re writing SQL. Even PostgreSQL, which has an excellent SQL parser, has been tripped up when encountering a user table.  Just use plural names and you’re far less likely to have a collision.  Don’t name the ID column “id”  This is a sin I’ve been guilty of for years. When working with a client in Paris, I had a DBA complain when I named my id columns id and I thought he was being pedantic. After all, the customers.id column is unambiguous, but customers.customer_id is repeating information.  And later I had to debug the following:  SELECT thread.* FROM email thread JOIN email selected ON selected.id = thread.id JOIN character recipient ON recipient.id = thread.recipient_id JOIN station_area sa ON sa.id = recipient.id JOIN station st ON st.id = sa.id JOIN star origin ON origin.id = thread.id JOIN star destination ON destination.id = st.id LEFT JOIN route ON ( route.from_id = origin.id AND route.to_id = destination.id ) WHERE selected.id = ? AND ( thread.sender_id = ? OR ( thread.recipient_id = ? AND ( origin.id = destination.id OR ( route.distance IS NOT NULL AND now() &gt;= thread.datesent + ( route.distance * interval '30 seconds' ) )))) ORDER BY datesent ASC, thread.parent_id ASC  Do you see the problem? If the SQL had used full id names, such as email_id , star_id , and station_id , the bugs would have stood out like a sore thumb while I was typing out this SQL , not later when I was trying to figure out both what I did wrong and why I don’t drink as much as I should.  And by request of a few people who couldn’t see the errors, here’s the SQL after it’s corrected. It’s very clear that “star_id” and “email_id”, or “station_id” and “station_area_id” are probably not valid comparisons. As previously mentioned, if SQL had a decent type system, this SQL would not even have compiled.  SELECT thread.* FROM email thread JOIN email selected ON selected.email_id = thread.email_id JOIN character recipient ON recipient.character_id = thread.recipient_id -- station_area_id = character_id is probably wrong JOIN station_area sa ON sa.station_area_id = recipient.character_id -- station_id = station_area_id is probably wrong JOIN station st ON st.station_id = sa.station_area_id -- star_id = email_id is probably wrong JOIN star origin ON origin.star_id = thread.email_id JOIN star destination ON destination.star_id = st.star_id LEFT JOIN route ON ( route.from_id = origin.star_id AND route.to_id = destination.star_id ) WHERE selected.email_id = ? AND ( thread.sender_id = ? OR ( thread.recipient_id = ? AND ( origin.star_id = destination.star_id OR ( route.distance IS NOT NULL AND now() &gt;= thread.datesent + ( route.distance * interval '30 seconds' ) )))) ORDER BY datesent ASC, thread.parent_id ASC  Do yourself a favor and use full names for IDs. You can thank me later.  Column Naming  As much as possible, name columns very descriptively. For example, a temperature column doesn’t make sense for this:  SELECT name, 'too cold' FROM areas WHERE temperature &lt; 32;  I live in France and for anyone here, 32 would be “too hot”. Instead, name that column fahrenheit .  SELECT name, 'too cold' FROM areas WHERE fahrenheit &lt; 32;  Now it’s completely unambiguous.  Also, when you have foreign key constraints, you should name the columns on each side of the constraint identically, if possible. For example, consider this perfectly reasonable, sane, SQL.  SELECT * FROM some_table s JOIN some_other_table o ON o.owner = s.person_id;  That looks sane. There’s really nothing wrong with it. But when you consult the table definition, you discover that some_other_table.owner has a foreign key constraint against companies.company_id . That SQL is, in fact, wrong. Had you used identical names:  SELECT * FROM some_table s JOIN some_other_table o ON o.company_id = s.person_id;  Now it’s immediately clear that we have a bug and you can see it on a single line of code and don’t have to go consult the table definition.  However, it should be noted that this isn’t always possible. If you have a table with a source warehouse and a destination warehouse, you might want a source_id and a destination_id to compare with your warehouse_id . Naming them source_warehouse_id and destination_warehouse_id will make this easier to follow.  It should also be noted that in the example above, owner is more descriptive of the intent than company_id . If you feel this is likely to cause confusion, you can name the column owning_company_id . That can still embed the intent of the column in the name while giving you a strong hint as to its intent.  Avoid NULL Values  Saving the best (or is it worst?) for last! This is a tip that many experienced database developers are aware of, but sadly, it doesn’t get repeated often enough: don’t allow NULL values in your database without an excellent reason.  This will take a bit of time because this is an important, but somewhat complicated topic. First, we’ll discuss the theory, then we’ll discuss their impact on database design, and we’ll finish up with a practical example of the serious problems NULL values cause.  Database Types  In the database, we have various data types , such as INTEGER , JSON , DATETIME , and so on. A type is associated with a column and any value added should conform to the type associated with that column.    But what’s a type? A type is a name, a set of allowed values, and a set of allowed operations. This helps us avoid unwanted behavior. For example, in Java, what happens if you try to compare a string and an integer?  CustomerAccount.java:5: error: bad operand types for binary operator '&gt;' if ( current &gt; threshhold ) { ^ first type: String second type: int  Even if you can’t see by glancing at the code that current &gt; threshhold is comparing incompatible types, the compiler will trap this for you.  Ironically, databases, which store your data—and are your last line of defense against data corruption—are terrible at types! I mean, really, really bad at them. For example, if your customers table has an integer surrogate key, you can do this:  SELECT name, birthdate FROM customers WHERE customer_id &gt; weight;  That, of course, doesn’t make a lick of sense and in a sane world would be a compile-time error.   Many programming languages make it trivial to trap type errors like this but databases make it hard.  But that’s not how databases generally behave, quite possibly because when the first SQL standard was released in 1992  , computers were slow beasts and anything that complicated the implementation would undoubtedly have made databases slow.  And here’s where the NULL value comes into all of this. There is one place where the SQL standard got this right and that’s with the IS NULL and IS NOT NULL predicates. Since the NULL value is, by definition, unknown, you can’t possibly have operators defined for it. That’s why IS NULL and IS NOT NULL exist instead of = NULL and != NULL . And any NULL comparison results in a new NULL value.  If that sounds strange, it becomes much easier if you say “unknown” instead of NULL :   NULL Unknown comparisons result in NULL unknown values.   Ah, now it makes perfect sense!  What does a NULL value mean?  Now that we have the tiniest amount of type theory under our belt, let’s examine the practical implications.  You need to pay a $500 bonus to all employees who earn more than $50K a year in salary. So you write the following SQL.  SELECT employee_number, name FROM employees WHERE salary &gt; 50000;  And you just got fired because your boss earns more than $50K but their salary isn’t in the database (their employees.salary column is NULL ) and the comparison operator can’t compare a NULL with 50000.  And just why is that value NULL ? Maybe their salary is confidential. Maybe the information hasn’t arrived yet. Maybe they’re a consultant and they’re not paid a salary. Maybe they’re paid hourly and are not salaried. There are plenty of reasons why that data might not be available.  The existence or non-existence of data in a column suggests that it depends on something other than just the primary key and your database is possibly denormalized. Thus, columns which might have NULL values are good candidates for creating new tables. In this case, you might have tables for salaries , hourly_rates , none_of_your_business and so on. You’ll still get fired for blindly joining on salaries and missing that your boss doesn’t have one, but at least your database is starting to present you with enough information to suggest that there’s more to the problem than just a salary.  And yes, this was a silly example, but it leads to the final nail in the coffin.  NULLs lead to logical impossibilities  You might have some sense that I’m being a bit pedantic about NULL values, but we have one final example and it’s caused much real-world grief.  Years ago I was in London working for a domain registrar and trying to figure out why a somewhat messy 80 line SQL query was returning incorrect data. There was a particular case where data absolutely should have been returned, but wasn’t. I’m embarassed to say that it took me about a day to track it down and it was a combination of a few things:   I had used an OUTER JOIN  Those can easily generate NULL values  NULL values can cause your SQL to give you incorrect answers   That last statement is something most database developers are unaware of, so let’s look at an example Database In Depth  by C.J. Date. First, a trivial schema with two tables.   suppliers    supplier_id city  s1 London    parts    part_id city  p1 NULL     Those should be pretty clear and it’s harder to get a simpler example.  The following, of course, returns p1 .  SELECT part_id FROM parts;  But what about the following?  SELECT part_id FROM parts WHERE city = city;  That returns no rows since you cannot compare a NULL value with anything—not even another NULL and not even the same  NULL . That seems odd because the city for every given row must the be the same city, even if we don’t know it, right? And that leads us to the following. What does this return? Try to work out the answer before reading it below.  SELECT s.supplier_id, p.part_id FROM suppliers s, parts p WHERE p.city &lt;&gt; s.city OR p.city &lt;&gt; 'Paris';  We get no rows because we can’t compare a NULL city ( p.city ) and thus neither branch of the WHERE clause can evaluate to true.  However, we know that the unknown city either is Paris or it is not Paris . If it’s Paris, the first condition is true ( &lt;&gt; 'London' ). If it’s not Paris, the second condition is true ( &lt;&gt; 'Paris' ). Thus, the WHERE clause must be true, but it’s not, leading to SQL which generates logically impossible results.  That was the bug which bit me in London. Any time you write SQL which can generate or include NULL values you run the risk of having SQL lie to you. It doesn’t happen often, but when it does, it’s devilishly hard to track down.  Summary   Use underscore_names instead of CamelCaseNames  Table names should be plural  Spell out id fields ( item_id instead of id )  Don’t use ambiguous column names  When possible, name foreign key columns the same as the columns they refer to  Add NOT NULL to all column definitions, when possible  Avoid writing SQL which can generate NULL values, when possible   While not perfect, the above database design guidelines will make your database world a better place.  ","title":"Database Design Standards","url":"\/articles\/database-design-standards.html"},{"body":"   The Constraints  Getting Started  The Tyranny of ORMs  Caching  SOAP  Tax Server  The Big Test  What We Delivered  Overview  New Caching Architecture  Data Storage Performance  Implement Staged Processing  Process Streamlining  Repeatable  Caveat     The Devil went down to Georgia. He was lookin' for a soul to steal. He was in a bind 'cause he was way behind. He was willing to make a deal. \"Devil Went Down To Georgia\"—The Charlie Daniels Band   Our client had just won a nice contract but were in a bind. Their legacy codebase, while powerful, was slow. They could not process more than 39 credit card transactions per second. They needed to get to 500 transactions per second for an event lasting 30 minutes. Because the event was highly publicized, there was a tight deadline. They had two weeks to get a proof of concept running, improving their performance by an order of magnitude.  They turned to our company, All Around the World  , because we have a proven track record with them.  We had a senior systems architect, Shawn, and a senior software architect, Noel, on the project. Our managing director, Leïla, oversaw the project and ensured that if we had questions, she had answers. I was brought in from another project because there was simply too much work to do in two weeks. Fortunately, though I didn't know the project, Shawn and Noel knew the system well and helped get me up to speed.  The Constraints  There were several key constraints we had to consider. Our overriding consideration was ensuring that PCI-compliance (Payment Card Industry compliance) was strictly adhered to to ensure that customer data was always protected. Second, the client had developed an in-house ORM (object-relational mapper) many years ago and like all projects, it grew tremendously. While it was powerful, it was extremely slow and had a lot of business logic embedded in it. Third, because we only had two weeks for the first pass, we were given permission to take \"shortcuts\", where necessary, with the understanding that all work was to be thoroughly documented and tested, and easy to either merge, remove, or disable, as needed.  Finally, we could change anything we wanted so long as we didn't change the API or cause any breaking changes anywhere else in the code. There was no time to give customers a \"heads up\" that they would need to make changes.  Getting Started    Source    Because the event only lasted 30 minutes, whatever solution we implemented didn't have to stay up for long. This also meant that whatever solution we implemented had to be disabled quickly if needed. We also knew that only a few customers would use this new \"fast\" solution, and only one payment provider needed to be supported.  Noel immediately started tracing the full code path through the system, taking copious notes about any behavior we would need to be aware of. Shawn was investigating the databases, the servers, the network architecture, and assessing what additional resources could be brought online and tested in two weeks.  I, being new to the project, started by taking a full-stack integration test representing one of these transactions and studied it to learn the code and its behavior. In particular, I wanted to better understand the database behavior as this is often one of the most significant bottlenecks.  I was particularly concerned because the client was in the midst of a large migration from Oracle to PostgreSQL, so making changes at the database level was not an option. We needed to know immediately if this was one of the bottlenecks. I wrote code which would dump out a quick summary of database activity for a block of code. It looked sort of like this:    explain dbitrace( name =&gt; 'Full stack transaction request', code =&gt; sub { $object-&gt;make_full_request(\\%request_data) }, save =&gt; '\/tmp\/all_sql.txt', );  The summary output looked similar to this:  { report =&gt; { name =&gt; 'Full stack transaction request', time =&gt; '3.70091 wallclock secs' }, sql =&gt; { delete =&gt; 1, insert =&gt; 7, select =&gt; 137, update =&gt; 32, total =&gt; 177, } }  We had a problem. Even our client was surprised about the amount of database activity for a single \"buy this thing\" request.  None of the database activity was particularly slow, but there was a lot of it. Deadlocks weren't an issue given that the changed data was only for the current request, but the ORM was killing us.  The Tyranny of ORMs    How it feels to work on most in-house ORMs. Source    I love working with a good ORM, but ORMs generally trade execution speed for developer speed. You have to decide which is more important to you. Further, in two decades of working with ORMs, I have only once seen an in-house ORM which was on-par with, or superior to, commercial or open source products. And it was an ORM optimized for reporting, something many ORMs struggle with.  For our client's ORM, every time a request was made it would gather a bunch of metadata, check permissions, make decisions based on whether or not it was using Oracle or PostgreSQL, check to see if the data was cached, and then check to see if the data was in the database. Instantiating every object was very slow, even if there was no data available. And the code was creating&mdash;and throwing away without using&mdash;hundreds of these objects per request.  We considered using a \"pre-flight\" check to see if the data was there before creating the objects, but there was so much business logic embedded in the ORM layer that this was not a practical solution. And we couldn't simply fetch the data directly because, again, the ORM had too much business logic. We had to reduce the calls to the database.  After an exhaustive analysis, we discovered several things.   Some of the calls were for a \"dead\" part of the system that no one really remembered but was clearly unused.  Numerous duplicate calls were being made to unchanging data. We could cache those objects safely.  A number of calls were being made for important data that wasn't relevant to our code path, so we could skip them.   Our first step at addressing the problem was to ensure that everything was wrapped in configuration variables that allowed us to easily turn on or off different code paths for our project. Fortunately, the client had a system that allowed them to update the configuration data without restarting the application servers, so this made our choice much safer.  Once that was in place, our first pass cut our SQL calls roughly in half, tremendously sped up the code, and the tests still passed. But we were nowhere near 500 transaction per second.  Caching  It's said that the three most common bugs in software are cache invalidation and off-by-one errors. And we had caching problems in spades.  We couldn't afford cache misses during the event, so we needed to \"preheat\" the caches by ensuring all information was loaded on server startup. Except that we had a few problems with this.  First, it was slow because there was a lot of data to cache. If we needed to rapidly bring new servers online during the event, this was not an option.  Second, it consumed a lot of memory and we were concerned about memory contention issues. Disk swapping was not an option.  Third, the app caches primarily used per-server memcached instances, but the data was global, not per-server, thus causing a lot of unnecessary duplication. Shawn already knew about this, so one of the first things he did was set up a secure Redis cluster with failover to replace memcached, where appropriate. Since we only had to heat the cache once, bringing up servers was faster, and we significantly reduced per-server memory consumption.    Now, where did I put that data? Source    The application itself also used heavy in-memory caching (implemented as hashes), which we replaced with pre-loaded shared cache entries, thereby lowering memory requirements even further.  As part of this work, we centralized all caching knowledge into a single namespace rather than the ad-hoc \"per module\" implementations. We also created a single configuration file to control all of it, making caching much simpler for our client. This is one of the many features we added that they still use today.  SOAP    Possibly not the most original image I've come up with ... Source    Another serious bottleneck was their SOAP server. Our client made good use of WSDL (Web Services Description Language) to help their customers understand how to create SOAP requests and SOAP was integral to their entire pipeline. The client would receive a SOAP request, parse it, extract the necessary data, process that data, create another SOAP request, pass this to back-end servers, which would repeat the process with the new SOAP request.  SOAP stands for \"Simple Object Access Protocol.\" But SOAP isn't \"objects\", and it's not \"simple.\" It's a huge XML document, with an outer envelope telling you how to parse the message contents. Reading and writing SOAP is slow . Further, our client's SOAP implementation had grown over the years, with each new version of their SOAP interface starting with cutting-and-pasting the last version's code into a new module and modifying that. There were, at the time of the project, 24 versions,   most of which had heavily duplicated code. To optimize their SOAP, we were facing a problem. However, we dug in further and found that only one version would be used for this project, so our client authorized us to skip updating the other SOAP versions.  We tried several approaches to fine-tuning the SOAP, including replacing many AUTOLOAD (dynamically generated) methods with static ones. In Perl, AUTOLOAD methods are optional \"fallback\" methods that are used when Perl cannot find a method of the desired name. However, this means Perl must carefully walk through inheritance hierarchies to ensure the method requested isn't there before falling back to AUTOLOAD . This can add considerable overhead to a request. For this and other reasons, the use of AUTOLOAD is strongly discouraged.  Replacing these methods was extremely difficult work because the AUTOLOAD methods were used heavily, often calling each other, and had grown tremendously over the years. Many of them were extremely dangerous to pick apart due to their very complex logic. We managed to shave some more time with this approach but stopped before we tackled the most complicated ones. There was only so much risk we were willing to take.  Noel, in carefully reading through the SOAP code, also found several code paths that were slow, but didn't apply to our requests. Unfortunately, we could not simply skip them because the SOAP code was tightly coupled with external business logic. Skipping these code paths would invariably break something else in the codebase. What he proposed, with caution, is the creation of a special read-only \"request metadata\" singleton. Different parts of the code, when recognizing they were in \"web context\", could request this metadata and skip non-essential code paths. While singletons are frowned upon by experienced developers, we were under time pressure and in this case, our SOAP code and the code it called could all consult the singleton to coordinate their activity.  The singleton is one of the many areas where deadlines and reality collide. Hence, the \"Devil Went Down to Georgia\" quote at the beginning of this case study. We were not happy with this solution, but sometimes you need to sacrifice \"best practices\" when you're in an emergency situation. We alerted our client to the concern so they could be aware that this was not a long-term solution.  We also leveraged that singleton to allow us to alert the back-end that encrypted transaction data was available directly via Redis. The back-end merely needed to decrypt that data without the need to first deserialize SOAP and then decrypt the data. Another great performance improvement. Sadly, like many others improvements, it was specific to this one customer and this one payment provider.  However, this approach allowed us to remove even more SQL calls, again providing a nice performance boost.  Tax Server    In the 80s, I used to do accounting by hand. Source    By this time, with the above and numerous other fixes in place, we were ready to do our initial QA runs. We didn't expect to hit our 500 TPS target, but we were pretty confident we had achieved some major speed gains. And, in fact, we did have some pretty impressive speed gains, except that periodically some of our requests would simply halt for several seconds. Due to the complicated nature of the system, it wasn't immediately clear what was going on, but we finally tracked down an issue with the tax server.  Our client was processing credit card transactions and for many of them, taxes had to be applied. Calculating taxes is complicated enough that there are companies that provide \"tax calculation as a service\" and our client was using one of them. Though the tax service assured us they could handle the amount of traffic we were sending, they would periodically block. It wasn't clear if this was a limitation of their test servers that we would avoid with their production servers, but we could not take this chance.  We tried several approaches, including caching of tax rates, but the wide diversity of data we relied on to calculate the tax meant a high cache miss rate, not allowing us to solve the problem. Finally, one of our client's developers who knew the tax system fairly well came up with a great solution. He convinced the tax service to provide a \"one-time\" data dump of tax rates that would not be valid long but would be valid long enough for our short event. We could load the data into memory, read it directly, and skip the tax server entirely. Though the resulting code was complicated, it worked, and our requests no longer blocked.  The Big Test  With the above, and many other optimizations, we felt confident in what we delivered and in a late-night test run, our client was ecstatic. We were handling 700 transactions per second, almost twenty times faster than when we started. Our client hired us because they knew we could deliver results like this. Now it was time for our first production run, using beefier servers and a much faster network.  Cue the \"sad trombone\" music. We were only running about 200 to 300 transactions per second. As you may recall, the client had been migrating from Oracle to PostgreSQL. The QA servers were far less powerful than the production servers, but they were running PostgreSQL. The production servers were running Oracle. Oracle is the fast, beastly sports car that will quickly outpace that Honda Accord you're driving ... so long as the sports car has an expert driver, constant maintenance, and an expensive, highly trained pit crew to look after it.    How I felt after our first production test run. Source    Out of the box, PostgreSQL just tends to be fast. Often companies find that a single PostgreSQL DBA working with competent developers is enough to easily handle their traffic. But we had Oracle. And it wasn't going away before the deadline. Our first two weeks was up and our proof of concept worked, but greatly exceeding client requirements still wasn't fast enough.  However, this was good enough for the proof of concept, so it was time to nail this down. It was going to be more nights and weekends for us, but we were having a huge amount of fun.  Oracle wasn't the only problem, as it turns out. This faster, higher capacity network had switches that automatically throttled traffic surges. This was harder to diagnose and work around but alleviated the networking issues. Interestingly, it was Leïla who spotted that problem simply by listening carefully to what was being said in meetings. She has this amazing ability to hear when something isn't quite right.  With the use of Oracle being a bottleneck, we had to figure out a way to remove those final few SQL statements. Our client suggested we might be able to do something unusual and it's based on how credit cards work.    Source    When you check into a hotel and reserve your room with a credit card, your card is not charged for the room. Instead, they \"authorize\" (also known as \"preauthorize\") the charge. That lowers the credit limit on your card and reserves the money for the hotel. They will often preauthorize for over the amount of your bill to allow a small wiggle room for incidental charges, such as food you may have ordered during your stay.  Later, when you check out, they'll charge your card. Or, if you leave and forget to check out, they have a preauthorization, so they can still \"capture\" the preauthorization and get the money deposited to their account. This protects them from fraud or guests who are simply forgetful.  In our case, the client often used preauthorization for payments and after the preauthorization was successful, the card would be charged.  It worked like this: the front-end would receive the request and send it to the back-end. The back-end would submit the request for preauthorization to the payment provide and the payment provider would respond. The back-end sent the response to the front-end and if the preauthorization was denied, the denial would be sent back to the customer. However, usually it was approved and the front-end would then submit the charge back to the back-end.   The back-end would then submit the preauthorized charge to the payment provider and lots of network and database activity helped to keep the server room warm. The entire process was about as fast as a snail crawling through mud.  What the client pointed out is that the money was reserved, so we could charge the card after the event because we had already preauthorized it. Instead, we could serialize the charge request and store it in an encrypted file on the front-end server (with the decryption key unavailable on those servers). Later, a utility we wrote read and batch processed those files after the event, allowing us to finish the transaction processing asynchronously.  This worked very well and by the time we were done, our changes to the code resulted in a single SQL SELECT statement. Further, after this and a few other tweaks, we were easily able to hit 500 transactions per second on their production system. When the event happened shortly after, the code ran flawlessly.  What We Delivered    Source    Overview  Increase the performance of Business Transactions from 39\/sec to 500+\/sec.  The overall implemented solution could not change the outward facing API calls as end users of the service were not to be impacted by back-end changes. The solution could also not effect the traditional processing as clients would be issuing both traditional and streamlined transactions.  New Caching Architecture  Implement a new redundant shared caching architecture using Redis as a data store, to replace the existing on application server memcached instances.  Update existing application code to remove the use of internal application caches (implemented as hashes), and replace with pre-loaded shared cache entries.  The new caching system caches items using both the legacy and the new caching system, so that production systems can be transitioned to the unified caching system without downtime.  Allowed a cache storage back-end to define its storage scope, so that caches can be managed based on the storage locality:   \"process\": cache is stored in local per-process memory  \"server\": cache is stored on the local server, and shared by all processes on the server  \"global\": cache is stored globally, and shared by all processes on any server   Data Storage Performance  Create an additional data storage path that replaces storing intermediate data in the back-end shared database with the shared caching architecture.  Remove the use of a third party provider API call, and replace with a static table lookup.  Implement Staged Processing  As data processing in real time is not a requirement, add a data path that allows for separation of the front end API responses from the third party back-end processing. This allowed for heavy back-end processes to be deferred and processed in smaller batches.  Process Streamlining  Worked closely with the client to determine which parts of the process are required during the processing, and which can be bypassed to save overhead.  Code review and clean up. Refactor code base to remove extraneous or ill-performing legacy code.  Repeatable  The process was re-engineered to be configurable allowing the same pipeline to be reimplemented for future events\/customer requirement through simple documented configuration options, while not impacting traditional processing.  Caveat  This information was compiled from our reports during this project and not written while the project was ongoing. Some events may not have happened in the exact sequence listed above.  ","title":"Case study: 500 TPS","url":"\/articles\/project-500.html"},{"body":"    Introduction  Programming A New Feature  Use Control Panels  Don’t Use Control Panels  Permanent Features  “Sampled” Switches  Naming Conventions  Operations  Errors  Communications  Simplicity  Clean Features Regularly  Don’t Check Features at Compile=Time  Conclusion     Introduction  One of the best things to come out of the modern DevOps movement is the aggressive push for “feature switches” (also known as “feature flags” or “feature toggles”). At All Around the World  , we strongly recommend them for our clients and sometimes we implement them, but I’m still surprised that many companies don’t use them. They’re dead-simple to build and they’re a powerful tool for site reliability. Unfortunately, many articles discuss how to build them or use them, but best practices are neglected. So we’ll skip the implementation and instead focus on using feature switches effectively.  Note: the discussion below refers to “sites” because this is in the context of websites. Obviously, it doesn’t have to be.  A couple of decades ago when I was working for a small web-development firm in Portland, Oregon, we released a new version of our software that had a critical bug in a new feature. The bug was serious enough that it took down the entire web site. We had to revert to an earlier version (from CVS, the good ‘ol days!) and redeploy the web site. Our largest client’s site was down for an hour and that was not good. Had we used feature switches, we could simply have turned the feature off and the client might not have even noticed.  Note that you don’t need to implement everything I suggest in your first pass; you just need to get up and running and comfortable with feature switches. Once you do, you can better understand the additional tooling support needed for your use case.  Programming A New Feature  For the developer, using feature switches is straight-forward. Imagine that you always show a list of categories for products you sell, but you’re fetching it from the database every time. Developers argue for caching the list, but sometimes it changes and arguments over cache invalidation stall development. Finally, an agreement is reached, but you don’t want to roll the new functionality out blindly, so you use a feature switch. Here’s some pseudo-code in Perl.  my $categories; if ( feature_on('cached_categories') ) { $categories = $cache-&gt;category_names; } else { $categories = $app-&gt;category_names; }   Side note: the above is shown for simplicity. In reality, $app variables often imply some kind of global state and if it’s mutable, that can be dangerous, but we’re keeping it simple.   In the above, if the cached_categories feature is on, we fetch features from the cache. If it’s not, we fetch it from the database. There are numerous ways this can be done on the front-end, but we’ll skip those for the purposes of this article.  Why would you bother to use a feature switch for the above? You’ve tested it. The customer won’t see a difference. You know it works. Right?  No, you don’t. What if your cache invalidation doesn’t work quite as expected? What if your cache isn’t loaded? What if your brand-new caching system isn’t as reliable as you thought and you actually need the following?  my $categories; if ( feature_on('cached_categories') ) { $categories = $cache-&gt;category_names || $app-&gt;category_names; } else { $categories = $app-&gt;category_names; }  Sound paranoid? One definition of a good programmer is someone who looks both ways before crossing a one-way street. So you roll out your new code with your feature turned off and after you verify everything is good, you turn on your feature. How?  Use Control Panels  Typically, a good system for feature switches should have a web-based control panel that lists each feature and shows, amongst other things:   The name of the feature  A description of what the feature is for  The current on-off state  Whether the feature is permanent (more on that later)  An “on-off” history of the feature   The “on-off” history is simply an audit log. At minimum, you want:   Who toggled the feature  The date it was turned on\/off  Why it was turned on\/off  Possibly the release-tag of the software   An audit log of the history can be immensely useful if you want to turn on a feature that was turned off for several months and you don’t remember why.   For companies that make heavy use of feature switches, this can be invaluable. If your company is structured well enough, you might even want to include what team the feature belongs to, or what departments are responsible for it. Because so many companies are structured differently, it’s hard to give clear guidance without knowing the specifics.  Don’t Use Control Panels  Yes, feature switches are usually used via control panels and they make it easy to turn features on and off, but there are caveats. It sometimes happens that enabling a new feature causes every page on your site to crash, taking down the control panel! Oops. Having a control panel running on a dedicated server, separate from the servers using the features can help to mitigate this. Having command-line tools to turn features on or off can also be useful. Do not rely on control panels alone . Nothing’s worse than having everything web-based and losing web access.  Permanent Features  One thing often overlooked for features switches is that some of them should be permanent. Imagine that your site goes viral and you’re hitting unprecedented amounts of load and you’re struggling. Some sites handle this by having optional features on the site wrapped in permanent feature switches. If your site is struggling under a heavy load, being able to toggle off optional features such as autocomplete or “related searches” that aren’t critical can help you manage that load. Your site experience degrades gracefully instead of simply crashing.  “Sampled” Switches  Before turning on a switch, it’s often useful to turn it on only for a subset of requests. For example, you might only want 10% of users to experience it. If a feature turns out to be disastrous, this can limit the impact. However, this can be odd for someone who sees a new feature, refreshes a web page, and that feature disappears because they are no longer in the randomly selected subset. Making such features persistent per user can be tricky, though. Do you rely on cookies? Your mobile app may not use cookies. Or someone might switch browsers, such as switch from their work to their personal computer.  Don’t stress about this (unless you’re doing A\/B testing). “Sampled” features are short-lived. As feature switches are primarily a site reliability tool, the limited sample run is just there to avoid disaster and you should go 100% as soon as is feasible.  Naming Conventions  For one of our clients, they have no naming convention for their features, but they use features heavily. We wrote code to scan our CODEOWNERS file to dump a list of all features in our code in order to regularly monitor them. This allowed us to do heavy cleanup when we found many features that had been running successfully for years, but were never cleaned. Even this was limited because many of “our” features were scattered in other parts of the codebase.  To deal with this, establishing a naming convention early on can make it easier to search for features. Are you on the advertising team and you have a new feature offering a discount for first-time buyers? You use ADV- as a prefix: ADV-FIRST_TIME_BUYER . Searching for all features prefixed with ADV- gives you an overview of the features your team has running.  When you identify a naming convention, do not put ticket numbers in the feature name. Many companies switch ticketing systems and old ticket numbers become useless. Instead, having a “Ticket URL” field on your form when you create a feature allows others to click through and understand the intent of that feature.  Operations  Errors  If you dump out rich error data to error logs, include a list of running features, sorted by last start time. Frequently, new errors show up due to recently introduced features and this makes it much easier to debug, especially if first time buyers sometimes can’t place orders and you see the ADV-FIRST_TIME_BUYER feature was just switched on. This makes life much easier.  Communications  Let other teams know when you’re about to start a new features, especially your site reliability engineers (or equivalent). Give them a heads-up about the intent of the feature and what to look for if their are issues. Such communication can allow tracking down issues faster.  Simplicity  Feature switches are a site reliability tool. Keep them simple. You don’t want your site to break because someone made the feature switch tool too complicated. This is one of the reasons I prefer features to be “booleans” (in other words, “on or off”). Some recommend feature tools that allow you to do this:  my $categories; if ( 'enhanced' eq feature_on('SOME_FEATURE') ) { $categories = $app-&gt;categories_enhanced; } elsif ( 'expanded' eq feature_on('SOME_FEATURE') ) { $categories = $app-&gt;categories_expanded; } elsif ( feature_on('SOME_FEATURE') ) { warn &quot;Unknown feature category for SOME_FEATURE&quot;; } $categories \/\/= $app-&gt;category_names;  Does the above start to look complicated? Well, kinda. If you allow more than just “on” or “off”, you need to check for the case where the feature value doesn’t match expectations. Maybe someone added a new value to the feature? Maybe expanded was actually expandable and you misspelled it in your code. By allowing these multiple values, you’re complicating code that’s a site reliability tool. KISS: Keep it simple, stupid. Feature switches are there to make life easier. Don’t overthink them.  Clean Features Regularly  Aside from “permanent” features that you want to always be able to switch off, it’s important to regularly clean features. Otherwise, you get into messes like this:  if (feature_on('FEATURE_1')) { # do something if (feature_on('FEATURE_2')) { # do something else } elsif (feature_on('FEATURE_3') { # do yet another thing if (feature_on('FEATURE_4')) { ... } } }  Not only does that make the code harder to read, you can wind up with cases where features work well on their own, but there could be some unknown interaction between conflicting features that won’t be noticed unless you have the right combination of features toggled on or off. This is miserable to debug.  Don’t Check Features at Compile=Time  This one is a subtle trap. Feature switches are designed to allow you to quickly turn a feature on or off while the system is still running. However, if your code that checks the feature does so too early and not at runtime, you may find that you can’t turn your feature on or off! Here’s an example in Perl:  package Some::Package; sub import ($class) { if ( feature_on('MY_COOL_FEATURE') ) { ... } } # elsewhere package Another::Package; use Some::Package; ...  In the above, the use Some::Package statement will cause Some::Package to call import and that will check the feature switch, but that will happen only once. Toggling the feature switch later will likely not affect the code while Another::Package is resident in memory.  Worse, if some other code is loaded later , but the feature switch has been toggled, you may find that different parts of your codebase do not agree on whether or not a feature is running. This is not a good place to be.  Conclusion  Feature switches are an incredibly powerful site reliability tool which can often eliminate the need to redeploy your software if something goes wrong. This can improve your customer experience and dramatically cut costs. However, like any new system you implement, you want to take some time up front deciding how it will be used and what your best practices are.  You will probably find that the best practices you develop for your company differ from what I describe above. That’s OK. I’m not dogmatic. But planning for this in advance can give you greater confidence in pushing new code that might otherwise be cause for concern.  If you’d like top-notch experienced developers with decades of experience to come in and make your life easier, let me know . Our work at All Around the World  , is focused on pragmatic development custom-tailored to client needs.  ","title":"Feature Switch Best Practices","url":"\/articles\/feature-switch-best-practices.html"},{"body":"   Full disclosure: I just found out that because of my contributions to significant open source projects, I get access to Github Copilot for free. That being said, even before I realized that, I was using the free trial and I was impressed.   A few years ago, a client contracted with us to build out a system where they’d be automating many common tasks their people did. However, it was a political minefield. The employees thought management was trying to automate them out of jobs, so they were not always cooperative. Management insisted they were trying to free up employee time for more complex tasks, but the employees were suspicious.  Today’s AI revolution is having similar effects. People are losing their jobs to ChatGPT. Companies are turning to Stable Diffusion and other AI systems to generate images. And now Github Copilot  is here to write code for us.  I tend to be a late adopter of software. It’s not that I’m a Luddite. I just prefer to focus on building things rather than learning the hot new technology du jour. But my explorations into ChatGPT taught me pretty quickly that if you’re willing to accept the quirks, AI can be an amazing tool. So I decided to give Copilot a try.  I use vim. If you have a different setup, you’ll need to hit your favorite search engine for instructions.  First, you’ll need neovim or a relatively recent vim and install the Github Copilot vim plugin  . I confess that after two decades with vim, and having a highly customized vim setup, I was a bit nervous about whether or not Copilot was going to play well with my setup. There were a few slight bumps in the road.  First, I wanted to make sure Copilot was only active in certain filetypes. I added the following to the bottom of my ~\/.vimrc :  let g:copilot_filetypes = { \\ 'gitcommit': v:true, \\ 'markdown': v:true, \\ 'yaml': v:true, \\ 'perl': v:true \\ }  That gives me Copilot in git commit messages, markdown files, YAML files, and Perl.  If you have extremely large files, you may want to disable Copilot in those  :  autocmd BufReadPre * \\ let f=getfsize(expand(&quot;&lt;afile&gt;&quot;)) \\ | if f &gt; 100000 || f == -2 \\ | let b:copilot_enabled = v:false \\ | endif  I was pretty sure that my auto-completion hack wouldn’t work, so I disabled that first:  &quot; function! CleverTab() &quot; I need to be cleverer &quot; if strpart( getline('.'), 0, col('.')-1 ) =~ '^\\s*$' &quot; return &quot;\\&lt;Tab&gt;&quot; &quot; else &quot; return &quot;\\&lt;C-N&gt;&quot; &quot; endfunction &quot; inoremap &lt;Tab&gt; &lt;C-R&gt;=CleverTab()&lt;CR&gt;  Yeah, I should use a plugin for that, but I’m lazy. It worked for me.  Then I started playing around with Copilot, but for some reason, sometimes it would show me the code it was suggesting, but when I hit “tab”, it wouldn’t insert it. It was frustrating, until I remembered I had this in my ~\/.vimrc :  noremap &lt;leader&gt;pp :set invpaste&lt;cr&gt;  I’m often cutting-n-drooling things into my editor, so I hit ,pp to toggle paste mode. When I realized that, I knew I just needed to hit ,pp again to let Copilot work its magic.  I’ve seen quite a few people talk about how Copilot is going to put programmers out of work, but I was pretty sure that while it could write simple code, it wasn’t going to be able to write code for complex business logic. I was sort of wrong. Copilot quickly picked up my style and if I named things intelligently, it often offered perfect suggestions. In fact, a few times it was offering suggestions that were better than what I thought of at the time. Humbling.  Of course, like ChatGPT and friends, Copilot is prone to hallucinations. It’s tried to write documentation for methods that don’t exist. It’s tried to write code calling methods that don’t exist. Sometimes it writes code that works, but has nothing to do with what I need. However, so long as I’m vigilant, I can catch those errors. And sometimes, it’s just a matter of hitting “tab” a few times to get the code I want.  This morning, I was working on an immutable ORM I’m building for a client. At one point, Copilot added the following comment to my code:  METHOD: foreach my $method (keys %methods) { my $new_method = $self-&gt;metadata('columns')-&gt;{$method}; if ( $method eq $new_method ) { # don't install methods if they already exist next METHOD; ...  I chuckled at comment. It was code being used to auto-generate methods and of course I wanted to install those methods.  And then I read the comment again. That next METHOD; meant I’d be skipping that method. I had a bug in my code and Copilot caught it for me. It was a silly, obvious bug. I was knee-deep in a huge change that impacted 27 files and tiny bugs like this are easy to miss. Thank goodness for tests, but doubly-thank goodness for Copilot.  Did I say “humbling” already? Yeah, humbling.  And then what blew me away happened this weekend. I was porting the OpenSky API  from Python to Perl. What I often do when porting code is to take the code and paste it into my editor as a comment and follow along while porting. I was using the following Python:  def get_flights_from_interval(self, begin, end): &quot;&quot;&quot; Retrieves data of flights for certain time interval [begin, end]. :param int begin: Start of time interval to retrieve flights for as Unix time (seconds since epoch). :param int end: End of time interval to retrieve flights for as Unix time (seconds since epoch). :return: list of FlightData objects if request was successful, None otherwise. :rtype: FlightData | None &quot;&quot;&quot; if begin &gt;= end: raise ValueError(&quot;The end parameter must be greater than begin.&quot;) if end - begin &gt; 7200: raise ValueError(&quot;The time interval must be smaller than 2 hours.&quot;) params = {&quot;begin&quot;: begin, &quot;end&quot;: end} states_json = self._get_json( &quot;\/flights\/all&quot;, self.get_flights_from_interval, params=params ) if states_json is not None: return [FlightData(list(entry.values())) for entry in states_json] return None  As soon as I typed sub get_flights_from_interval , Copilot suggested this:  sub get_flights_from_interval ( $self, $begin, $end ) { if ( $begin &gt;= $end ) { croak 'The end time must be greater than or equal to the start time.'; } if ( ( $end - $begin ) &gt; 7200 ) { croak 'The time interval must be smaller than two hours.'; } my %params = ( begin =&gt; $begin, end =&gt; $end ); my $route = '\/flights\/all'; my $response = $self-&gt;_get_response( route =&gt; $route, params =&gt; \\%params ) \/\/ []; if ( $self-&gt;raw ) { return $response; } return OpenSky::API::Flights-&gt;new($response); }  The only change I made to it was rewriting the error messages.  There are a two interesting this about this. One, it worked perfectly the first time. Two, notice that $self-&gt;raw ? That’s something I added to other methods to return the raw data structures instead of objects. Copilot knew to add it there.  Also, the Python code returns None if it doesn’t find anything, meaning that you always have to remember to guard your code with an if check. My code always returns an OpenSky::API::Flights object, but with an empty iterator if there are no results. If you forget an if check, the consuming code will still work. If you need to check if you have results, you can still call $flights-&gt;count to see if anything is there. Copilot knew to add that, too.  Copilot is not only suggesting code, but accommodating my style.  To be fair, there are serious concerns about Copilot. It’s trained on code from a variety of sources, including code that is copyrighted. There’s an ongoing class action lawsuit as a result of this  .  But rather go into a number of issues with Copilot, I’ll point to the elephant in the room: developers are going to lose jobs because of this. This technology is in its infancy and it’s already writing code that’s better than many developers I’ve met. As the technology matures, more and more developers will find that simply writing good prompts will take care of much of their work for them. Already I’m finding that Copilot is increasing my productivity, but at the same time, I need to be on guard for subtle bugs that it might introduce. Since I’m not writing the code, I risk not thinking about it as much as I should. Fortunately, I’m fanatical about testing, so this limits the risk.  But as more companies shift to tools like this, they’ll be looking at senior developers like myself, developers with decades of experience, and they’ll be asking themselves why they’re paying us so much when Copilot can do the work for them. Junior devs might be finding it easier to get work since they don’t charge as much money and I think it’s currently unknown if the quality of our software will improve as a result.  I don’t know what the future holds, but one part is clear: we’re getting closer to being able to describe our software and having it written for us.  Note: the OpenSky code I wrote using Copilot is available on github  and the CPAN  . I’ve already had one bug report (my bug, not Copilot’s) and I fixed it and released this morning, along with more tests.  ","title":"Using Github Copilot with Vim","url":"\/articles\/using-github-copilot-with-vim.html"},{"body":"  You may recall that one Jack Sweeney created ElonJet  , a service to track Elon Musk’s jet in real time. This led to his account being suspended by Twitter, but you can find him on Mastodon  .  He has a public repository for tracking planes  , but it’s written in Python. I wanted to do the same for Perl, so I did. I wrote WebService::OpenSky  (also available on GitHub  ), which is a Perl wrapper around the OpenSky Network API  . From OpenSky Network’s home page  :   The OpenSky Network is a non-profit community-based receiver network which has been continuously collecting air traffic surveillance data since 2013. Unlike other networks, OpenSky keeps the complete unfiltered raw data and makes it accessible to academic and institutional researchers. With over 30 trillion ADS-B, Mode S, TCAS and FLARM messages collected from more than 5000 sensors around the world, the OpenSky Network exhibits the largest air traffic surveillance dataset of its kind. The mission of our non-profit association is to support open global air traffic research by universities and other not-for-profit institutions. You can find more information on our blog  .   I won’t go into great detail about the API, but it’s pretty simple. You can read the docs to understand more, but here’s a quick example:  use WebService::OpenSky; my $musks_jet = 'a835af'; my $openapi = WebService::OpenSky-&gt;new; my $days = shift @ARGV || 7; my $now = time; my $then = $now - 86400 * $days; my $flight_data = $openapi -&gt;get_flights_by_aircraft( $musks_jet, $then, $now ); say &quot;Jet $musks_jet has &quot; . $flight_data-&gt;count . &quot; flights&quot;;  As of this writing, that prints out: Jet a835af has 3 flights  The $flight_data object allows you to iterate over the flights, but it’s pretty limited. I wanted more. I wanted something like this:  Flight #1. Departed Monday, 29 May, 10:42 AM UTC from Austin-Bergstrom International Airport, Austin, Texas, US Flight #1. Arrived Monday, 29 May, 04:34 PM UTC at Ted Stevens Anchorage International Airport, Anchorage, Alaska, US Flight #2. Departed Thursday, 01 Jun, 04:32 AM UTC from unknown airport, unknown city Flight #2. Arrived Thursday, 01 Jun, 04:45 PM UTC at Austin-Bergstrom International Airport, Austin, Texas, US Flight #3. Departed Thursday, 01 Jun, 04:32 AM UTC from unknown airport, unknown city Flight #3. Arrived Thursday, 01 Jun, 04:45 PM UTC at unknown airport, unknown city  As you can see, the data itself is spotty, but the older the data, the more accurate it appears to be. So how did we get there?  First, we need to get the ICAO 24-bit hexadecimal code  for Musk’s private jet. It turns out he has three of them, but I’m only tracking the one he’s using the most. A quick search for “icao 24 Elon Musk’s jet” revealed this Hacker News discussion of it  . Various other sources confirmed this, with the N628TS registrations having the icao number of a835af .  Next, we need to get the flight data. The OpenSky API allows you to get flight data and that’s shown in the script above. However, it’s not very useful. I wanted to see the actual airport names and times (the times returned by the API are Unix epochs  ).  Getting the ICAO codes for the airports was a bit harder. Since I’m doing this open source, I eventually tracked down a fairly active GitHub repository with airport data  . I wanted to use the Geo::ICAO  module to convert the ICAO data, but it’s out-of-date and missing a lot of airports. Thus, I wrote this code:  sub get_icao_data { my $url = 'https:\/\/raw.githubusercontent.com\/mborsetti\/airportsdata\/main\/airportsdata\/airports.csv'; my $ua = Mojo::UserAgent-&gt;new; return eval { my $response = $ua-&gt;get($url)-&gt;res; my %lookup; if ( $response-&gt;is_success ) { my ( $fh, $filename ) = tempfile(); print {$fh} $response-&gt;body; close $fh; my $data = csv( in =&gt; $filename, headers =&gt; &quot;auto&quot;, ); %lookup = map { $_-&gt;{icao} =&gt; $_ } @$data; } \\%lookup; }; }  If any airports are missing from that (and it appears there may be some issues), I fall back to Geo::ICAO . The solution has been working well so far. Obviously, paid services are more accurate, but this is good enough for my needs.  I’ll skip explaining the rest of the code, but here’s a link to the full script  . Running that as perl lights.pl --days 30 outputs:  Flight #1. Departed Sunday, 07 May, 08:39 PM UTC from Miami-Opa Locka Exec Airport, Miami, Florida, US Flight #1. Arrived Sunday, 07 May, 11:01 PM UTC at Austin-Bergstrom International Airport, Austin, Texas, US Flight #2. Departed Monday, 08 May, 06:40 PM UTC from Austin-Bergstrom International Airport, Austin, Texas, US Flight #2. Arrived Monday, 08 May, 07:13 PM UTC at Corpus Christi International Airport, Corpus Christi, Texas, US Flight #3. Departed Monday, 08 May, 07:47 PM UTC from Corpus Christi International Airport, Corpus Christi, Texas, US Flight #3. Arrived Monday, 08 May, 08:21 PM UTC at Austin-Bergstrom International Airport, Austin, Texas, US Flight #4. Departed Monday, 08 May, 08:59 PM UTC from Austin-Bergstrom International Airport, Austin, Texas, US Flight #4. Arrived Monday, 08 May, 09:29 PM UTC at Corpus Christi International Airport, Corpus Christi, Texas, US Flight #5. Departed Monday, 08 May, 09:55 PM UTC from Corpus Christi International Airport, Corpus Christi, Texas, US Flight #5. Arrived Monday, 08 May, 10:10 PM UTC at Farm Services Inc Airport, Rio Hondo, Texas, US Flight #6. Departed Tuesday, 09 May, 02:07 PM UTC from unknown airport, unknown city Flight #6. Arrived Tuesday, 09 May, 05:19 PM UTC at Napa County Airport, Napa, California, US Flight #7. Departed Tuesday, 09 May, 06:37 PM UTC from Napa County Airport, Napa, California, US Flight #7. Arrived Tuesday, 09 May, 07:06 PM UTC at Norman Y Mineta San Jose International Airport, San Jose, California, US Flight #8. Departed Friday, 12 May, 06:25 AM UTC from Norman Y Mineta San Jose International Airport, San Jose, California, US Flight #8. Arrived Friday, 12 May, 07:15 AM UTC at Jack Northrop Field\/Hawthorne Municipal Airport, Hawthorne, California, US Flight #9. Departed Saturday, 13 May, 04:18 AM UTC from Jack Northrop Field\/Hawthorne Municipal Airport, Hawthorne, California, US Flight #9. Arrived Saturday, 13 May, 05:27 AM UTC at unknown airport, unknown city Flight #10. Departed Sunday, 14 May, 10:27 PM UTC from unknown airport, unknown city Flight #10. Arrived Monday, 15 May, 07:38 AM UTC at Paris-Le Bourget Airport, Paris, Ile-de-France, FR Flight #11. Departed Monday, 15 May, 06:19 PM UTC from Paris-Le Bourget Airport, Paris, Ile-de-France, FR Flight #11. Arrived Tuesday, 16 May, 03:48 AM UTC at Seidel Ranch Airport, Elroy, Texas, US Flight #12. Departed Friday, 19 May, 01:10 PM UTC from Austin-Bergstrom International Airport, Austin, Texas, US Flight #12. Arrived Friday, 19 May, 03:59 PM UTC at Norman Y Mineta San Jose International Airport, San Jose, California, US Flight #13. Departed Saturday, 20 May, 06:35 AM UTC from Norman Y Mineta San Jose International Airport, San Jose, California, US Flight #13. Arrived Saturday, 20 May, 11:12 AM UTC at Rhode Island Tf Green International Airport, Providence, Rhode Island, US Flight #14. Departed Saturday, 20 May, 07:54 PM UTC from unknown airport, unknown city Flight #14. Arrived Saturday, 20 May, 11:17 PM UTC at Seidel Ranch Airport, Elroy, Texas, US Flight #15. Departed Sunday, 21 May, 12:12 AM UTC from Austin-Bergstrom International Airport, Austin, Texas, US Flight #15. Arrived Sunday, 21 May, 02:45 AM UTC at Los Angeles International Airport, Los Angeles, California, US Flight #16. Departed Sunday, 21 May, 06:03 PM UTC from Los Angeles International Airport, Los Angeles, California, US Flight #16. Arrived Sunday, 21 May, 08:32 PM UTC at Seidel Ranch Airport, Elroy, Texas, US Flight #17. Departed Tuesday, 23 May, 02:41 PM UTC from Austin-Bergstrom International Airport, Austin, Texas, US Flight #17. Arrived Tuesday, 23 May, 05:38 PM UTC at Norman Y Mineta San Jose International Airport, San Jose, California, US Flight #18. Departed Thursday, 25 May, 03:40 PM UTC from Norman Y Mineta San Jose International Airport, San Jose, California, US Flight #18. Arrived Thursday, 25 May, 04:27 PM UTC at Jack Northrop Field\/Hawthorne Municipal Airport, Hawthorne, California, US Flight #19. Departed Friday, 26 May, 01:08 AM UTC from Jack Northrop Field\/Hawthorne Municipal Airport, Hawthorne, California, US Flight #19. Arrived Friday, 26 May, 01:55 AM UTC at Norman Y Mineta San Jose International Airport, San Jose, California, US Flight #20. Departed Friday, 26 May, 07:00 AM UTC from Norman Y Mineta San Jose International Airport, San Jose, California, US Flight #20. Arrived Friday, 26 May, 09:37 AM UTC at Seidel Ranch Airport, Elroy, Texas, US Flight #21. Departed Monday, 29 May, 10:42 AM UTC from Austin-Bergstrom International Airport, Austin, Texas, US Flight #21. Arrived Monday, 29 May, 04:34 PM UTC at Ted Stevens Anchorage International Airport, Anchorage, Alaska, US Flight #22. Departed Thursday, 01 Jun, 04:32 AM UTC from unknown airport, unknown city Flight #22. Arrived Thursday, 01 Jun, 04:45 PM UTC at Austin-Bergstrom International Airport, Austin, Texas, US Flight #23. Departed Thursday, 01 Jun, 04:32 AM UTC from unknown airport, unknown city Flight #23. Arrived Thursday, 01 Jun, 04:45 PM UTC at unknown airport, unknown city  As you can see, he arrived in Paris on Monday, 15 May, 2023 at 07:38 AM UTC. He spent a few hours there before flying back to the Seidel Ranch airport in Texas (that was the first airport which let me know that Geo::ICAO data was incomplete). A quick search revealed he was in Paris to meet with French president Emmanuel Macron.  Have fun!  ","title":"Tracking Elon Musk's Plane with Perl","url":"\/articles\/tracking-elon-musks-plane-with-perl.html"},{"body":"  The more I dig into the logic of Perl, the more I find that it uses constructive (intuitionist) instead of classical logic. But Perl developers think in terms of the latter, not the former. This causes us to often reason incorrectly about our software.     For example, if we have this:  if ( $x &lt; 3 ) { ... } else { ... }  We tend to think that $x is a number and $x &gt;= 3 holds in the else block. That might be true in a strongly typed (uses classical logic) language, but that’s not necessarily true in either Perl or constructive logic.  Here’s a counter-example in Perl:  my $x = {}; # a reference to a hash if ($x &lt; 3) { say &quot;if&quot;; } else { say &quot;else&quot;; }  That will print else because my $x = {}; say $x + 0; will often print a number like 4966099583. That’s the address of $x .  You won’t even get a warning.  Not useful.  So the else branch in Perl simply says the previous case didn’t hold, but it says nothing about the actual value of the number.  We can assert that $x is an integer and die if it’s not, but many don’t bother with that.  Classical versus Constructive  In classical logic, there’s the law of the excluded middle. Something is either true or it’s not. In constructive logic, things are true, not true, or other (unknown).  If $x is an integer, we know that it’s either 3 or it’s not, but we might not know it yet .   Note: Constructive (intuitionist) logic isn’t a mere mathematical curiosity. The well-known physicist Nicolas Gisin has written a short, easy-to-read paper showing how intuitionist logic might prove that there really is an “arrow of time” in physics  .   I find the idea of constructive logic useful enough that I wrote a Perl module named Unknown::Values  .  I proposed adding this new, three-value logic, or 3VL (true, false, unknown), to Perl and while there was some interest, some practical problems reared their ugly heads and the idea went nowhere.  Interestingly, one of the counterpoints was based on the confusion between classical and constructive logic. Complementing undef with a new unknown keyword (that salary might return):  # using undef values: my @overpaid = grep { defined $_-&gt;salary &amp;&amp; $_-&gt;salary &gt; $cutoff } @employees; # same thing with unknown values: my @overpaid = grep { $_-&gt;salary &gt; $cutoff } @employees;  As you can see, using unknown values instead of undef values makes the code shorter and easier to read. unknown salaries would not pass the grep. You don’t have to remember the defined check. It just works . Of course, you can run it in the other direction:  my @underpaid = grep { $_-&gt;salary &lt;= $cutoff } @employees;  And again, unknown values wouldn’t pass the grep. If you need those employees:  my @unknown_salary = grep { is_unknown $_-&gt;salary } @employees;  But this was presented as a counter-argument:  if ( $x &lt; 3 ) { ...; } else { ...; }  With 3VL, surely the else would also have to be skipped because we don’t know if the condition is false? Thus, we could have an if\/else block where the entire construct is skipped. That would be confusing madness!  Except that in 3VL, if the if condition doesn’t hold, the else can still fire because it represents an unknown value, as shown in the $x = {} example above.  The counter-argument might have worked for Java:  if ( someVar &lt; 3 ) { System.out.println(&quot;Less Than&quot;); } else { System.out.println(&quot;Not Less Than&quot;); }  In the above, if you hit the else block, you know both that:   someVar is a number.  someVar is not less than three.   Thus, Java follows classical logic. A statement is either true or false. There is no middle ground. Perl follows constructive logic, even though we tend to think (and program) in terms of classical logic.  Ignoring the type, for Perl this is more correct:  if ( $x &lt; 3 ) { ...; } elsif ( $x &gt;= 3 ) { ...; } else { ...; # something went wrong }  That’s because until we explicitly make a positive assertion about a value, we cannot know if a statement is true or false. Thus, in Perl, an else block is frequently not a negation of the if condition, but a catch block for conditions which haven’t held.  Again, Perl developers (and most developers using languages with dynamic types) tend to think in terms classical logic when, in fact, we’re using constructive logic. It usually works until it doesn’t.  How Can We Fix This?  brian d foy found my Unknown::Values  module interesting enough that he felt it should be in the Perl core:  Unknown::Value from @OvidPerl looks very interesting. These objects can&#39;t compare, do math, or most of the other default behavior that undef allows. This would be awesome in core. https:\/\/t.co\/xmD8yoEjIK &mdash; brian d foy (@briandfoy_perl) December 17, 2021   His observations match my concerns with using undef in Perl. However, when I proposed this to the Perl 5 Porters  , while some found it interesting, there were some serious concerns. In particular, Nicholas Clark wrote  (edited for brevity):   For this:  $bar = $hash{$foo};  If $foo happens to be unknown , is $bar always unknown ? Or is it undef if and only if %hash is empty?  That behaviour is arguably more consistent with what unknown means than the “always return unknown”.  ...  Does trying to use an unknown value as a file handle trigger an exception? Or an infinite stream of unknowns on read attempts?  What is the range [$foo .. 42] where $foo is unknown?  I think that most logically it’s an empty list, but that does seem to end up eliminating unknown-ness. Hence if we have first class unknowns, should we be able to have arrays of unknown length?   Ovid; It has a high-value win in eliminating common types of errors we currently deal with   And massive risk in introducing a lot of surprises in code not written to expect unknowns, that is passed one within a data structure.  Basically all of CPAN.  ...  There are about 400 opcodes in perl. I suspect that >90% are easy to figure out for “unknown” (for example as “what would a NaN do here?“) but a few really aren’t going to be obvious, or end up being trade offs between conceptual correctness and what it’s actually possible to implement.   Needless to say, this pretty much shot down the idea and these were annoyingly fair points.  In other words, leaking this abstraction could break a lot of code and people will be confused. For example, if DBIx::Class  were to suddenly return unknown instead of undef for NULL values, while the semantics would adhere much more closely to SQL, the behavior would be very suprising to Perl developers receiving a chunk of data from the ORM and discovering that unknown doesn’t behave the same way as undef .  So how would we deal with this? I can think of a couple of ways. Each would use the feature pragma to lexically scope the changes.  The first way would be to change undef to use 3VL:  use feature 'unknown'; my @numbers = ( 1, 2, undef,5, undef,6 ); my @result = grep { $_ &lt; 5 } @numbers; # result now holds 1 and 2  In this approach, any undefined value would follow 3VL. However, if you returned that undef outside of the current lexical scope, if falls back to the current 2VL (two-value logic: true or false).  However, we might find it useful (and easier) to have distinct unknown and undef behavior:  use feature 'unknown'; my @numbers = ( 1, 2, unknown,5, undef,6 ); my @result = grep { $_ &lt; 5 } @numbers; # result now holds 1 and 2, and `undef`  This would require unknown to behave like undef if it left the current lexical scope.  In other words, ensure that the developer who chooses to use 3VL has a tightly controlled scope.  But what do we do with the case of using unknown values as a filehandle or the keys to a hash or the index of an array?  $colors[unknown] = 'octarine';  Currently, for the Unknown::Values module, we have the “standard” version which mostly returns false for everything So sort and boolean operations ( $x &lt; 3 ) are well-defined, but almost anything else is fatal. (There is a “fatal” version which is fatal for just about anything, but I don’t think it’s very useful).  Why Not use Null Objects?  When Piers Cawley  first suggested that I use Null Object pattern  instead of unknown values, I balked. After all, you could have this:  my $employee = Employee-&gt;new($id); # not found, returns a null object if ( $employee-&gt;salary &lt; $limit ) { ... }  That puts us right back where we started because when salary returns undef , it gets coerced to zero (probably with a warning) and the &lt; $limit succeeds. That’s not the behavior we want.  Or maybe salary returns the invocant to allow for method chaining. Well, that fails because $employee-&gt;salary &lt; $limit silent coerces salary to the address of the object (thanks, Perl!) and that’s probably not what we want, either.  That’s when I realized an old, bad habit was biting me. When presented with a solution, I tend to look for all of the ways it can go wrong. I need to look for all of the ways it can go right. The more Piers made his case, the more I realized this could make sense if I can use the operator overloading of Unknown::Values . I could write something like this (a bit oversimplified):  package Unknown::Values::Instance::Object { use Moose; extend 'Unknown::Values::Instance'; sub AUTOLOAD { return $_[0] } }  Now, any time you call a method on that object, it merely returns the object, When you eventually get down to comparing that object, the overloading magic in the base class always causes comparisons to return false and we get the desired 3VL behavior.  We could go further and assert the type of the Unknown::Values::Object and do a -&gt;can($method) check and blow up if the method isn’t there, but that’s probably not needed for a first pass.  But there’s more! In the Wikipedia article about null objects  , they use an example of a binary tree to show how Null objects sometimes need overloaded methods. Using syntax from the Corinna OOP project for Perl  :  class Node { has $left :reader :param { undef }; has $right :reader :param { undef }; }  We can recursively calculate the size of the tree:  method tree_size() { # this recursive method adds 1 to $size for every node # found return 1 + $self-&gt;left-&gt;tree_size + $self-&gt;right-&gt;tree_size; }  But the child nodes might not exist, so we need to manually account for that:  method tree_size() { my $size = 1; if ( $self-&gt;left ) { $size += $self-&gt;left-&gt;tree_size; } if ( $self-&gt;right ) { $size += $self-&gt;right-&gt;tree_size; } return $size; }  That’s not too bad, but if we need to remember to check if the right and left are there every time we use them, sooner or later we’re going to have a bug.  So let’s fix that (note: this example is simplified since I’ll need to create a new version of Null objects just for Corinna).  class Null::Node :isa(Null::Object) { # break the recursion method tree_size () { 0 } }  class Node { use Null::Node; has $left :reader :param { Null::Node-&gt;new }; has $right :reader :param { Null::Node-&gt;new }; }  Because we’ve overridden the corresponding methods, our tree_size() doesn’t need to check if the nodes are defined:  method tree_size() { return 1 + $self-&gt;left-&gt;tree_size + $self-&gt;right-&gt;tree_size; }  In this simple case, this may overkill, but if we start to add a lot of methods to our Node class, we will have a Null object as default, using 3VL, and we don’t have to worry about building in extra methods that we don’t need. (But see the criticism section about null objects for more information  ).  Unknown::Values with Null object support is on the CPAN  and github  .  Caveat  Be aware that the latest version ( 0.100 )of Unknown::Values has backwards-incompatible changes  . In particular, any attempt to stringify an unknown value or object is fatal. That is a huge win because it protects us from this:  $hash{$unknown_color} = 'octarine'; $array[$unknown_index] = 42;  Both of those are clearly bugs in the context of unknown values, so they’re now fatal.  This doesn’t address all of Nicolas' concerns, but it definitely goes a long way to handling many of them.  ","title":"Constructive Versus Classical Logic in Perl","url":"\/articles\/constructive-versus-classical-logic-in-perl.html"},{"body":"    Why Leave Oracle?  Time Has Not Been Kind to Oracle  Added Benefits of PostgreSQL  Starting the Process  Success Criteria  SWOT Analysis  Developing a Migration Plan  1. Audit  2. Planning  3. Migration  4. Testing  5. Adjusting  6. Deployment  7. Rollback  Success!       The glory days before databases. Source    Why Leave Oracle?  Oracle is the fast, beastly sports car that will quickly outpace that Honda Accord you’re driving ... so long as the sports car has an expert driver, constant maintenance, and an expensive, highly-trained pit crew to look after it. Source   A couple of decades ago I was working for a company that was using Oracle on Linux. At the time, this was unusual, though becoming more common. It was a time when open source software was still looked upon with mistrust, so while we told our investors we used Oracle, we didn’t trumpet the fact that we used Linux.  One day we discovered a problem. An Oracle SELECT statement was sometimes returning different results. The query in question was extremely complex, but the results should not have changed as the underlying data was static. After much debugging, the company gave up trying to solve the issue and called Oracle. We were informed that while Oracle could run on Linux, it needed to be Red Hat Advanced Server.  So we switched to Red Hat Advanced Server. And the bug was still there.  &ldquo;No, you need be on version X of Red Hat Advanced server.&rdquo;  So we switched to version X of Red Hat Advanced Server. And the bug was still there.  &ldquo;No, you need be on version X, point release Y, of Red Hat Advanced server.&rdquo;  So we switched to version X, point release Y of Red Hat Advanced Server. And the bug was still there.  &ldquo;That’s a known bug. We&rsquo;re working on it.&rdquo;  Words were said. They were not nice words.  Time Has Not Been Kind to Oracle  In the early 2000s, switching from Oracle to PostgreSQL was a gamble. Today we’ve had several clients making the switch or in the process of it, but what holds them back is no longer “can we trust PostgreSQL?“, it’s “our entire technology stack assumes Oracle.”  And the driver behind the switch? It’s invariably Oracle’s insane costs with no clear return on investment for most companies. But there are plenty of technical reasons, too. Oracle DDL is not transaction safe.   Oracle violates the ANSI standard by often treating empty strings as NULL , causing all sorts of subtle bugs. And Oracle was built in an ASCII world, not a Unicode one, and it shows, painfully.  Added Benefits of PostgreSQL  First, PostgreSQL is generally chosen as the migration target due to its strong compatibility with Oracle and its excellent data handling. Fortunately, it’s now easy to find PostgreSQL admins (as opposed to Oracle admins) in the wild because PostgreSQL is both open source and free, so it’s much easier for a would-be DBA to get experience with the stack. In fact, one of our DBAs commented:   I never met anyone who knew how to use Oracle RAC (and debug its problems) unless they had already endured the hell of a broken Oracle RAC installation at least once ... on someone’s production server.   Cloud support for PostgreSQL is fantastic, with Amazon, Azure, and Google all supporting PostgreSQL.  And because PostgreSQL is SQL-92 (and increasingly, SQL-99 compliant) with an open extension library, its geospatial tooling and JSON support are arguably much stronger than Oracle’s. If you need those tools, you save a ton of money and get a better product.  Starting the Process  As with any project of this scope, you need to start out with a solid plan and that plan starts with a problem statement clearly outlining:   The problem you are trying to solve.  Why you want to solve the problem.  What your proposed solution is.  Measurable success criteria.  A SWOT analysis (or something comparable).   While points 1 through 3 are the most straightforward, it’s worth considering unusual ways of meeting your goals. For example, merely contacting Oracle and explaining that you’re considering leaving Oracle for a more affordable solutions might get enough of a discount on licensing that you don’t feel the immediate pressure to migrate. High-profile companies, or organizations that hook people young (such as universities) can often command hefty discounts.  You may also find that hiring an expert to review licensing is worthwhile. For example, hot standby servers need licensing, but cold standby with minimal use may not. Or you may find yourself paying for too many licenses because you have separate databases which can be merged, or you may find yourself paying for too many CPUs when a more powerful server may be a more cost-effective solution.  There are also many edge cases to consider. One company we know uses Oracle for a public facing application so that they can reassure investors that “they use Oracle”, but use PostgreSQL exclusively for everything internal.  Or there are the cases where clients use Oracle Enterprise applications but find many internal cases where they’re unnecessarily spending on Oracle licenses. The Enterprise applications are often difficult to extricate from, while individual applications still eat up enormous costs for no valid reason.  Success Criteria  However, it’s points 4 and 5 above (success criteria and SWOT analysis) that we find many clients don’t address. For example, the success criteria needs to be measurable and you should be able to address:   Performance  Functional requirements  Schedule  Budgeting  Resource allocation  Minimized disruption   All of those will need flexibility when you encounter surprising obstacles to the migration. Resource allocation is particularly important if you are partway through a migration and find yourself maintaining Oracle and PostgreSQL systems simultaneously. This is where having a dedicated team to oversee the process is helpful because they are focused on successful delivery and are less likely to be pulled away on internal “distractions” that arise throughout the migration process.  SWOT Analysis  A SWOT analysis  is simply a listing of Strengths, Weaknesses, Opportunities and Threats. For example, a clear strength of moving to PostgreSQL is the significant cost savings. Weaknesses may include poor internal knowledge of PostgreSQL. Opportunities can involve the custom data types and DDL-safe transactions, while threats might involve a need for write-heavy multi-master replication.  The above weaknesses or threats can be addressed, but you need to understand them and plan for them up front, with a signicant focus on mitigating weaknesses and threats.  Developing a Migration Plan  The migration plan that you develop will need to be fine-tuned for how your company operates, so a “one-size fits all” plan is not possible to develop. However, a successful migration plan should generally include the following steps:   Audit  Planning  Migration  Testing  Adjusting  Deployment  Rollback   1. Audit  The audit stage starts with identifying every area in your organization where Oracle is used, identifying all licenses and expirations (you may need to renew during the migration process), and how Oracle is being used. We’ve found many clients are merely using Oracle as a data store and use few, if any, stored procedures or functions. This is underutilizing the power of the database but has the advantage of simplifying migrations.  The software audit is a critical portion of the overall audit. Many applications using sophisticated ORMs may find they need few, if any, changes when switching from Oracle to PostgreSQL. However, other applications—particularly those building SQL directly—will need a deeper audit to understand how the migration is to be accomplished.  Further, because the migration process is time-consuming, make sure you’re only migrating what needs to be migrated. Do you have historical data that doesn’t need migration? Do you have tables or stored procedures which are no longer used? You may find it simpler to skip those.  2. Planning  With the primary audit finished, it’s time to develop a detailed plan on migrating to PostgreSQL. We know of one company in Portland, Oregon, which managed to get 90% of their software tests on a large application passing in a single evening. More typical, however, is the case where the “Oracle-only” assumption runs deep. This is where PostgreSQL shines, because its SQL is mostly compatible with Oracle’s.  The planning for most common use cases involves the schema migration, data migration, and software migration. If you have a small schema (and this is almost always a case where PostgreSQL is preferable to Oracle), the schema migration can be done by hand. However, there are many open source and commercial tools which can migrate an Oracle schema to a PostgreSQL schema. Care will still need to be taken, however, especially if you have Unicode data.  3. Migration  The actual migration at the database level will take some time, but it in the software level that you’ll particularly experience grief. Having a robust test is imperative, as is a strong QA department.  One issue we routinely encounter is Unicode. Unicode is problematic for Oracle because it was developed in an ASCII world while PostgreSQL fully understands and supports Unicode. For example, in Oracle, depending on the value of NLS_LENGTH_SEMANTICS , you may find that VARCHAR2(10) holds 10 characters or 10 bytes. If it’s bytes, you cannot store 日本 , because that’s actually twelve bytes! In PostgreSQL, VARCHAR(10) is guaranteed to hold 10 characters , not bytes , so 日本 is just fine. (Note: you can also use NVARCHAR2 in Oracle to enforce character semantics, but NVARCHAR2 assumes a maximum of two-bytes per character  , so you still can’t store 日本 in an NVARCHAR2(10) field).  Another common pitfall is that Oracle often treats the empty string as NULL . This is in violation of the ANSI-SQL standard and causes no end of headaches for developers. For example, in Oracle you can often find cases of outer joins with ambiguous semantics because you don’t know if field you’re asking for was missing (not joined), or is the empty string.  Or one surprising issues we’ve stumbled across: Oracle allows you to declare integers ( INT ), but internally it’s a NUMBER(38,0) and conversion tools generally have NUMERIC(38,0) in the PostgreSQL version of the schema. While there are those who insist there’s no performance difference between the two  , when ported to PostgreSQL, you should use the appropriate integer datatypes and not their NUMBER(p,0) equivalents. We’ve found significant performance improvements by converting NUMERIC(p,0) types to appropriate integer types.  There are many other subtle differences ( TIMESTAMP WITH TIME ZONE , join differences, transactions, function incompatibilities, and so on). The free, open source Orafce  tool module will make much of the pain go away, but not all (and it may prove to be a crutch you shouldn’t rely on). Again, comprehensive automated testing backed by a QA department is helpful here.  4. Testing  While it’s improved dramatically over the past decade or two, testing is the bane of many IT companies. Developers tend to test that their software does what they want it to do, while strong QA teams try to test that their software doesn’t do what they don’t want it to do. The difference is subtle, but it’s the difference between creation and destruction.  A key example comes from your issue and bug tracking systems. If you find that certain bugs are recurrent, this is often the case of a developer fixing the general issue for a specific case. A strong QA department should recognize the general nature of the problem and extend their tests to cover the general issue, not the specific one.  While this problem is not specific to an Oracle to PostgreSQL conversion, it is an issue which can mean the difference between a successful and unsuccessful migration. You will need to be sure that your testing approach is solid and that you’re not just testing that everything works with valid inputs, but that they fail in an expected way with invalid inputs.  But be aware that testing isn’t just about whether or not your software works as expected. Can you restore from a backup? How long does it take for you to failover? What’s your throughput like? Do you have “stealth” applications which assume Oracle and you’ve just switched? Perfectly functioning software doesn’t matter when your infrastructure catches fire.  And please keep in mind that this testing isn’t just a “one off” for the migration. A solid testing plan allows you to ensure that regressions don’t occur in the PostgreSQL implementation as it undergoes further development. Spending this time and money up front will save you time and money later.  5. Adjusting  The migration and testing aspects of your transition will likely need to be repeated more than once. There’s not much to say about the “adjustment” phase other than:   You must have measurable success criteria  Those criteria should be automated as far as is possible  Rejected work must document the success criteria and how to achieve them   However, keep in mind that you must not let the perfect be the enemy of the good. Are you excelling in some areas and under performing in others? Is this an acceptable trade off? Are all success criteria really that important? Remember: your primary goal is to facilitate the Oracle to PostgreSQL migration. You can take full advantage of PostgreSQL’s power later.  6. Deployment  Congratulations! Your migration and testing have worked. You’ve successfully met your success criteria. You’ve made adjustments here and there, but you’re ready to roll things out. At this point, it’s very hard to give specific advice because the details of the your deployment will vary widely from company to company.  This deployment should follow your standard patterns as much as possible to avoid confusion, but it needs to be planned carefully, with care taken to minimize downtime as much as possible (especially if your database-backed applications are public facing). However, because this deployment will likely be more significant than most, it does need to be set up in such a way that when obstacles are encountered that cannot be easily overcome, you can immediately abandon the deployment.  If you’re not using it yet, something like a blue-green deployment  approach is recommended here. Changing live systems is hard and it’s easy to paint yourself into a corner. Instead, build a duplicate of the system on production (containers work brilliantly here, of course), and run your tests against your changes. When you’re satisfied that all is well, point everything over to the new system, being ready to switch it back in a hurry, if needed.  And if you do this on a Friday, you deserve the pain you’re going to inevitably going to face.  7. Rollback  If you’ve followed the steps above and gotten to the point of rollout, you’re probably in a good place but you still need to plan for rollback. Will you need it? Probably not. But if you haven’t planned for it and you need it, it’s like having a fire without fire insurance. Discovering a critical production problem a week after deployment could involve losing a lot of data.  There are a few things to note about the rollback plan.   It must be thoroughly documented  It must be thoroughly tested  It is developed in concert with the migration and deployment plans  You must decide if data loss is acceptable or if data migration is part of the plan   The last point is very tricky. If you have something catastrophic happen, you may want to accept data loss rather than risk having to rollback your rollback because you couldn’t migrate new data to the old system. If a rollback is necessary, it’s like to be spotted right away and not a week later, so you’re unlikely to face this decision, but prepare for it.  Remember that a key flaw of release management is that people tend to expect success, not prepare for failure. By expecting success but planning for failure, you’ll put yourself in a great position.  Success!  Congratulations! You’ve successfully replaced Oracle with PostgreSQL. You’ve saved yourself a lot of money, you have a database that’s easier to manage, and you get to take advantage of many of the powerful features of PostgreSQL.  ","title":"Migrating from Oracle to PostgreSQL","url":"\/articles\/moving-from-oracle-to-postgresql.html"},{"body":"  As every developer learns, it’s easy to get naming wrong. For example, if you have a variable named temperature , but your system handles both imperial and metric, is that temperature in Fahrenheit or Celsius? Getting that wrong could lead to all sorts of problems. A similar situation exists for subroutine and method names. What does the method scale do? Who knows? If it’s named scale_size , you have a much better idea.  But let’s discuss predicate methods in object-oriented code. A predicate method is a method that returns true or false. In Ruby, you can end a method name with a question mark and by convention, it should return true or false.  class Point @@number_of_points = 0 def initialize(x,y) @x = x @y = y @@number_of_points += 1 end def self.too_many? return @@number_of_points &gt; 2 end end point1 = Point.new(3,3) point2 = Point.new(4,0) puts Point.too_many? # prints false point3 = Point.new(4,8) puts Point.too_many? # prints true  This is a lovely feature because writing if Point.too_many? reads like a question. However, most popular languages tend to only allow alphanumerics and underscores in method names. So in a recent code review for our free-to-play game, Tau Station ( Sign up here! It’s fun.  ), I called out a case where a predicate method was poorly named. Sadly, it was a predicate method I had written a long time ago and it’s a mistake I still make today. The following examples are made up, but keep to the spirit of the issue.  Quick: what does the following bit of code do? Make a guess. vip stands for “very important person” and means the character has used a VIP pack and they receive in-game benefits.  if ( $character-&gt;vip ) { # do something }  Does vip return how much time is left on their VIP status? Does it return a VIP pack? Does it return a boolean indicating if the character has VIP? We can rename that function to make it very clear.  if ( $character-&gt;is_vip ) { # do something }  Now it’s crystal clear what that means. Predicate functions should generally start with a verb that implies true or false: is_ , can_ , has_ , and so on. So it was with a bit of surprise on a later code review that I read two developers disagreeing about a predicate method name. One named it can_purchase and the reviewer was suggesting allow_purchase . They didn’t budge in their respective positions and both are very good developers, so I was asked to take a look.  In glancing at the code snippet, I immediately agreed that the predicate method should be named can_purchase . But after I wrote that on the review, I had doubts because the answer was too obvious. Why on earth would their be a dispute about this? I decided to dig further. The problem stemmed from an unfortunate feature provided by Moose  (an object-oriented framework for Perl) and perhaps a subtle misunderstanding of object-oriented code.  This is worth an explanation of the confusion.  The code in question was providing an API response for purchasing a visa to visit Gaule-affiliated stations  . The API returns actions you can take regarding a purchase. However, if the character meets certain conditions, the API call should provide additional information allowing a purchase of the visa, including its duration and price.  The code sort of looked like this:    package Veure::Model::API::Visa; use Veure::Moose; use Veure::Types qw(Bool); use Veure::Util::Data qw(:json); with qw(Veure::Model::API::Role::Composable); has can_purchase =&gt; ( isa =&gt; Bool, required =&gt; 1, ); sub _build_api_content ($self) { my $character = $self-&gt;viewer; my %api_response ( area_actions =&gt; { renew_visa =&gt; { available =&gt; json_false }, purchase_visa =&gt; { available =&gt; json_false }, }, ); if ( $self-&gt;can_purchase ) { # add additional data } return \\%api_response; } __PACKAGE__-&gt;_finalize;  That’s the code that I glanced at which made me think that can_purchase is a perfectly acceptable predicate method, until I realized the mistake I made.  Let’s take another look at can_purchase .  has can_purchase =&gt; ( isa =&gt; Bool, required =&gt; 1, );  In the Moose OO framework, the above creates a method named can_purchase . In our version of Moose, that method is read-only, it returns a boolean, and it’s required to be passed to the constructor. So far, so good.  However, the interface for the module is called in a very standard way:  return Veure::Model::API::Visa-&gt;new( viewer =&gt; $character, can_purchase =&gt; $can_purchase )-&gt;api_content;  The can_purchase data isn’t intended to be exposed to the outside world. The method should have been written like this:  has can_purchase =&gt; ( isa =&gt; Bool, reader =&gt; '_can_purchase', required =&gt; 1, );  I’ve written before about needing to keep interfaces small and here we were, violating that rule. Unfortunately, Moose, by default, provides public methods for all instance data and it gets annoyingly hard to avoid this. It’s annoying enough that I often don’t even call that out on code reviews.  This should not be a predicate method. Instead, this is a constructor argument that directs the instance on its behavior. It’s an order , giving a command to the instance. By convention in our code, we like to write our constructor arguments such that orders use active verbs to indicate what’s going on. This is why the reviewer suggested allow_purchase .    This might seem petty or trivial, but the distinction is important because not making said distinction can lead to subtle errors in reasoning. In this case, a predicate method returns a boolean indicating something about the state of the object. It’s only doing a single thing. But objects aren’t just about state. They’re state plus behavior . Predicates return information about the state after the behavior has occurred. This constructor argument tells the object to do something before the behavior has occurred. They look similar, but they’re not.  Because Moose makes everything public by default (because it’s hard not to do this in Perl), including creating methods for your instance data, it’s natural that can_purchase seems like the correct name since it’s public and asking if the character can purchase a visa. But the consumer is making an API calls and receives a data structure, not the instance itself. The data it receives tells the consumer what information to present, not a predicate method call.  I expect that when the Corinna OOP system  finally becomes core to Perl, much of this confusion will go away.  class Veure::Model::API::Visa :does(Veure::Model::API::Role::Composable) { use Veure::Util::API qw(format_price); use Veure::Util::Data qw(:json); field $allow_purchase :new; method build_api_content :private () { my %api_response ( area_actions =&gt; { renew_visa =&gt; { available =&gt; json_false }, purchase_visa =&gt; { available =&gt; json_false }, }, ); if ( $allow_purchase ) { # add additional data } return \\%api_response; } }  In the above, $allow_purchase is not exposed in the interface and it’s much clearer what its role is.  ","title":"Naming and Object-Oriented Code","url":"\/articles\/naming-and-object-oriented-code.html"},{"body":" Nothing profound in this post. It's just a bit of a rant.  As of this writing, there is an advertisement for a \"Senior Perl Developer\" that is part onsite in the Bay Area of the US. The salary listed works out to to about $15 an hour. For those unfamiliar with buying power in the US, $15 an hour is what many politicians are arguing should be the national minimum wage. This company is offering senior software developers a fry cook's pay in one of the most expensive locations in the United States.  At least they're up front about it. The worst job offer I received came as a shock from someone who owned a company well-known in their field, and a company I very much wanted to work for. My previous employer announced they were going out of business and I was scrambling to find work in a serious recession in Portland, Oregon.  I scored an interview with the owner of a small, but profitable consulting firm. They were working with some technologies I very much wanted to work with, but at the beginning of the interview, I discovered that most of their development was in C. I didn't mind that at all, but I confessed that I hadn't touched C in years. \"No problem,\" I was told. He knew my background, had solid recommendations, and was sure I would have no issues picking up C again.  OK, that's a good sign.  Later, he was showing me a propriety programming language they had devised, targeting a particular industry. While showing me code examples, I noticed something strange. I asked the him \"do you know what an SQL injection attack is?\"  As it turns out, they had been vulnerable for years but only discovered that two weeks earlier. He was impressed. And then I noticed their sessions apparently used sequential ids to determine the user, an even easier attack vector! And they had no tests covering this behavior! And, and, and ...  I was getting excited about the amount of value I could bring and wanted to dive in immediately just to fix all of the security holes.  Yes, the interview was definitely going well.  The interview starting covering personal ground and we started talking about politics, food we love, and other things. This was rather unprofessional but happened naturally. Turns out we had quite a few things in common.  I was definitely breathing a sigh of relief here. The dice just kept falling my way and I was feeling like the luckiest many alive.  Then the offer arrived.  It started with a massive pay cut, but given that I was going to learn several technologies I wanted to get into and I was about to be unemployed, I could deal with that, albeit reluctantly. I could pay my bills on it, but was definitely going to have to cut some expenses.  Then I found out that there was not only no paid vacation, there was no vacation.  Um, excuse me? \"We're a small company and we can't afford to have anyone take vacation. However, we need you to work overtime and when you accrue enough, we give you compensating time off.\"  The owner was shocked when I declined the offer. He thought the fact that we got along so well would count for something. Even when I pointed out the massive pay cut combined with overtime and no holidays, he said \"that's just how being a small company works.\"  Sadly, that is how much business works: how much can we get for how little? But there's no employee loyalty when they know you're screwing them. I stuck to my guns and found a much nicer position handling Big Data in Perl. Even when desperate, acting desperate is rarely a good idea.  ","title":"The Worst Job Offer","url":"\/articles\/the-worst-job-offer.html"},{"body":"    The first Corinna commits  What’s Next?  Perl 7  Perl 8  Installation Manager  Concurrency  Data Constraints  Miscellaneous  Conclusion     The first Corinna commits  Recently, some of the first commits of the new Corinna object system  for Perl have been merged into the Perl core. If you’re not familiar with it, I wrote a brief tutorial  .  Around the same time as those commits were merged, I was rummaging around on an old computer and discovered, to my shock, that I had an old implementation of what I was trying to do, before that fateful day at a conference when Sawyer X, the man driving Perl development, saw my work, liked it, but urged me to to stop the implementation and just design something great . (As an aside, it also doesn’t quite appear to match my previous descriptions of it. Oops).  With my initial commit of that code on September 2, 2018, the Corinna project has been roughly four and a half years from conception to the initial core implementation. Since that time, I worked on design, made an initial announcement, and led a design team of some of the top experts in Perl, with much feedback coming from others who’ve designed object systems such as Moose or Moo, and from that effort, Corinna was accepted and the Perl core will have clean, mature OOP system for the first time (and one whose prototype, Object::Pad  has been running in production at more than one company).  Keep in mind that bless and @ISA will still be there. We’re not taking anything away because the Perl 5 Porters (a.k.a., P5P) strive hard to not break backwards-compatibility.  If you’re curious, here’s what a simple 2D point class looked like;  class Point { has [qw\/x y\/] =&gt; ( default =&gt; sub { 0 } ); sub clear ($self) { $self-&gt;@{ 'x', 'y' } = ( 0, 0 ); } }  That’s not good at all, but I was trying to work within Perl’s limitations. It was also largely a response to improve upon Moxie  :  package Point { use Moxie; extends 'Moxie::Object'; has x =&gt; ( default =&gt; sub { 0 } ); has y =&gt; ( default =&gt; sub { 0 } ); sub x : ro; sub y : ro; sub clear ($self) { $self-&gt;@{ 'x', 'y' } = (0, 0); } }  I quite liked the intent of some of the ideas in Moxie, but that syntax is what pushed me to finally build Corinna. By contrast, here’s the same code in Corinna:  class Point { field ( $x, $y ) :param = 0; method clear () { ( $x, $y ) = ( 0, 0 ); } }  There will be an adjustment period while people learn a new way to program in Corinna, the same way that learning a new programming requires learning to think a different way, but overall, the Perl community is very happy about the project. Some are not, but that’s OK. Having a perfectly homogeneous community, marching in lockstep, no questions asked, would be a creepy thing. It would also be a useless thing. Diversity of opinion is not only healthy, but needed.  What’s Next?  That’s a great question. I’ve co-written one book on Perl and written another (linked in the images at the top of the page), I served two decades at The Perl Foundation , the last several years as a member of the board, I have a bunch of modules on the CPAN  , and our company, All Around the World  , is still doing well, with multiple contracts. I’m currently having fun building an OpenAPI  system using Dancer2  (something I’ve not had much chance to play with before), and another company inquired about help from us to go in and fix their object-oriented mess, including cleaning up a bunch of old code to use Moose  .  So we’re doing fine, but whither Perl? Or to abuse Shakespeare:  To Perl or not to Perl, that is the question: Whether 'tis nobler in the code to suffer The slings and arrows of outrageous criticism, Or to add commits against a sea of troubles, And by rebasing, end them? To die(), to sleep(), No more; and by a sleep() to say we end The heartache and the thousand natural shocks That Perl is heir to. 'Tis a consummation Devoutly to be wished. To die(), to sleep(); To sleep–perchance to dream–ay, there's the 'no strict.'  Sue me. I like poetry. Why do you think my nickname is “Ovid”?  Let’s be honest, Perl isn’t as popular as it used to be. There’s a lot of angst in the Perl community expressed over this. There’s also a lot of finger-pointing. There are many reasons that Perl is not as popular as it once was, but rather than ruminate on history and stir up debates over things we cannot change, I want to look to the future.  Fix the problem, not the blame.  So what’s the problem?  Sure, we can talk about fewer Perl developers or fewer companies starting new projects in Perl, but those are symptoms . Instead, I want to focus on one problem that Perl has and I think it’s both important and solvable.  Perl 5 was released in October of 1994 and there’s a misconception that Perl hasn’t changed much since then. I still remember a Linux conference in Prague where a programmer was telling me that Perl hasn’t been updated in decades and this is a widespread misconception that hurts Perl (we suck at marketing).  Just looking at the stable releases in the last decades:        Pumpking  Version  Release Date      Ricardo  5.16.0  2012-May-20    Ricardo  5.18.0  2013-May-18    Ricardo  5.20.0  2014-May-27    Ricardo  5.22.0  2015-Jun-01    Ricardo  5.24.0  2016-May-09    Sawyer X  5.26.0  2017-May-30    Sawyer X  5.28.0  2018-Jun-22    Sawyer X  5.30.0  2019-May-22    Sawyer X  5.32.0  2020-Jun-20    Sawyer X  5.34.0  2021-May-20    Ricardo  5.36.0  2022-May-27     As you can see, Perl gets a regular, major release every year. Some of the new features introduced into Perl include native try\/catch blocks. Function signatures are now on by default. Unicode 14.0 is supported. True booleans are supported. Some older misfeatures, such as indirect object syntax and multidimensional are disabled (they’re still available, but if you don’t know what they are, don’t worry). There are some amazing improvements to the regular expression engine, widely considered the best in the programming world  .  There’s much more, but many issues are very Perl-specific and even many Perl programmers are not aware of the work done by the porters. The Perl 5.36.0 release alone represents approximately a year of development since Perl 5.34.0 and contains approximately 250,000 lines of changes across 2,000 files from 82 authors (taken from the perldelta file for 5.36.0).  Note that 82 developers might seem like a lot, but it’s also incomplete. Many people who’ve contributed haven’t done so by committing directly to the Perl repository.  So Perl is still under heavy development, but it’s hard to argue that we’ve done a great job communicating that, so how can we communicate that?  Perl 7  The excellent Raku Programming Language  was known as “Perl 6” for many years, but it diverged far enough from its Perl roots to merit a separate name. So when there was much excitement about a new release of Perl called Perl 7, there was great news coverage of it  .  In short, even with a fresh Perl install, to use Perl effectively today, you often get the behavior of an older version of Perl and you have to ask for the modern features. If you’re new to Perl, there’s good chance you wouldn’t know that.  Unfortunately, that work was halted amidst disagreements over direction and the Perl Steering Council (PSC) later released a discussion of about what it would take to get to Perl 7  . Is Corinna going to be part of Perl 7? The PSC recently announced that it might  . I think it should, but asking my opinion is like asking a parent if their newborn is cute, so you can ignore me.  But on the off chance you don’t want to ignore me (you’re reading this, right?), Perl 7 will likely be a newsworthy release, showing the world that Perl is still in active development, but the features in the language are rather marginal in value, in the eyes of many. Corinna is the single-largest change to the Perl language in three decades, and that might make a splash, but again, I’m biased. Nonetheless, releasing Perl 7 is not trivial and Corinna might not be the top priority of the developers who will actually get the work done. I can live with that.  At the risk of offending a few people, barring Corinna, any release of Perl 7 is likely to be much of what we have in Perl right now, but with easier defaults. It will simply make it easier to write Perl, but not offer much “new” to current developers, beyond a message to the programming world.  For technical and political reasons,   Perl 7 without Corinna might be exactly what Perl needs. But even then, Perl 7 will be a far cry from the Perl released back in 1994. Perl’s been evolving rapidly over the years, but without people hearing about it.  But how can we push this further? That’s what I’ve been thinking about.  Perl 8  Perl’s regular expression engine is world-class, but it’s hardly the reason to choose a language. Or to keep it. What else does Perl offer? With Corinna, Perl will have one of the nicest OOP systems of popular dynamic languages (er, I may be biased, but when I look at other “popular” dynamic languages, I wince). So that can be two killer features. But we need more, but to get to “more,” we need to address the fact that the Perl core is missing a few features that other languages provide. We need to fix that.  Installation Manager  Recently, I hit an issue with a client. My code was correct, but failed in the test environment due to a “Can’t locate object method” error. As it turns out, the actively maintained CPAN module I was using was last updated in 2022, but the version the client had was released over two decades ago! Worse, my attempt to locally debug this was frustrated by some particulars in the client setup which caused me to find a current local version of the module and not the archaic version the client had. Fortunately, it was a trivial fix, but spending time debugging this issue cost our client money they should not have had to spend.  Per the nodejs blog  :   CPAN and RubyGems have blurred the lines between development tools and system package managers. With npm we wish to draw a clear line: it is not a system package manager. It is not for installing firefox or ffmpeg or OpenSSL; it is for rapidly downloading, building, and setting up Node packages. npm is a development tool. When a program written in Node becomes sufficiently mature it should be distributed as a tarball, .deb , .rpm , or other package system. It should not be distributed to end users with npm .   Out of the box, Perl has a cpan client which lets you install Perl modules. It’s great for developing libraries, but we don’t offer great support for developing applications . When I cd into a client directory, I want to run something like perl-local . and have it pick up the version of Perl for this project and use this project’s CPAN modules . Having this completely isolated from other Perl projects which might have different needs. We have many Perl tools designed to solve various parts of the problem, but it’s time to put them into the core. There’s now a discussion on the P5P mailing list about addressing this  .  To put this as a simplistic user story:   As a developer  I want to easily set up multiple isolated Perl environments  So I can support multiple isolated clients\/projects  Because my work shouldn’t have the fragility of picking up the wrong dependencies   There’s a lot of work there, but a developer installing Perl for the first time should be able to solve this, just as developers downloading other languages for the first time can.  Concurrency  &lt;snark&gt;Promises, promises.&lt;\/snark&gt;  Much of the programming world today is dealing with concurrency in some form. There are serious problems this solves, including:   CPU-intensive work can be split across multiple CPUs (or GPUs!).  Don’t wait on code that might block due to external resource constraints.   P5P has long preferred that new ideas for Perl be tested on the CPAN before incorporating these ideas into the core. This allows us to see what is and isn’t working (or in some cases, just to better understand the syntax people prefer). There are multiple attempts to solve the concurrency issue on the CPAN and given that this feature is being used more and more, and becoming a default feature in many languages, I think the core needs to support his.  Data Constraints  As with an installation manager and concurrency, the CPAN offers many solutions for constraining your data. You might notice I avoided the word “type.” Computer scientists have reasonable disagreements about types and type systems. Computer programmers have shouting matches.  I know what I want from types. Others strongly disagree. However, a very workable compromise has emerged in Perl. This is largely based on Type::Tiny  and similar modules. For Moose, for example, I can define an attribute like this:  has expires_at =&gt; ( is =&gt; 'rw', isa =&gt; InstanceOf ['DateTime' ], );  With that, I can read and write the expires_at value and know that the value must be a DateTime instance or a subclass (er, or anything that overloaded the isa method to lie and say it’s a DateTime class, but we’ll ignore that because it opens up a fascinating discussion well beyond the scope of this post).  For a subroutine, I might do this:  sub reset_totals ($count, $totals) { state $check = compile(Int, HashRef[Int]); ($count, $totals) = $check-&gt;($count, $totals); $count = constrain($count); foreach my $key (keys $totals-&gt;%*) { $totals-&gt;{$key} = $count; } }  Unlike the attribute approach, the sub solution only guarantees the kind of data you have in variables at the time of the $check-&gt;() . In the above subroutine, if constrain returned an array reference instead of an integer, the above code would not only continue to run, but it would do so without warning, to the point of corrupting the data in the code calling this function!  Note that the above is not a hypothetical problem. It happens, but we kid ourselves that our test suite will protect us. In reality, attaching a data constraint to variables instead of statements would fix the above. TypeScript isn’t perfect, but I’ve enjoyed the fact that many of these errors simply don’t occur (though TypeScript handles this differently).  We are not, however, there yet. Type::Tiny and related tools are great ways of minimizing the issue, but their syntax is clumsy and their features are limited, largely because they’re working within the limitations of Perl. This is not a criticism of these modules, though. We have to start somewhere. However, our company  is finding more and more clients relying on Type::Tiny due to the safety it provides. More work is needed here.  Miscellaneous  I think Perl needs native exceptions. It needs native stack traces. Native constants, including deep structures. Copy-on-write would be useful, but it’s not clear how feasible this is. There’s a lot more, but what I’ve written might already be controversial enough that stopping is probably good.  Conclusion  We need a vision of Perl’s future. Many of the things we need in the Perl core are things we’ve re-implemented many times on the CPAN. Not having them in core means developers have to learn their current environment’s way of handling this, and makes it easier to choose a module that might be inappropriate, or worse, unmaintained. This increases the cognitive burden on all developers who, I’m sure, are more interested in building solutions than arguing over them.  Perl 7 is going to be wonderful, but it’s time to start thinking about Perl 8.  If you have ideas about what Perl needs for the future, please leave a comment below.  ","title":"The Future of Perl","url":"\/articles\/the-future-of-perl.html"},{"body":" The following is a true story, but with names changed.  When I work with clients to build software, I take the usual steps of understanding their needs, gathering requirements, learning about their customers, and so on. At this point I have a model on paper of roughly what the software is intended to do, so they get surprised when I immediately turn to database design.  \"Who care about database design? What about mockups? What about workflows?\"  Let me tell you about \"Bob's Luxury Goods.\" I worked for this company many years ago and they had a retail store selling ... you guessed it ... luxury goods. They'd ask all customers for a billing address and if they had a different delivery address. At the database level, they had a \"one-to-many\" relationship between customers and addresses.  That was their first problem. A customer's partner might come into Bob's and order something and if the address was entered correctly it would be flagged as \"in use\" and we had to use a different address or deliberately enter a typo. Fortunately, addresses were case-sensitive, so many people had UPPER-CASE ADDRESSES.  We should have had a many-to-many relationship between customers and addresses so we could handle the case where more than one person would share the same address, but we didn't. Further, I was never allocated the time to fix the database because it was \"cheaper\" to remove the restriction on \"flagged\" addresses and allow a duplicate address to be used.  Naturally, being a luxury goods company, we had many repeat customers and sometimes they would move and if we didn't find the duplicate address, or the address with the \"typo\", we might update the address for one partner, but not the other. That was a headache, but it didn't happen frequently enough for management to worry about it.  That's when the marketing department had a brilliant, inexpensive idea. You see, we periodically did mass mailings of special events to our customers. Since we had the software to do mass mailings, why not import a mailing list of all addresses in high net worth areas and mail everyone about upcoming special events? So the company went ahead and bought a database with all of these addresses, but forgot to mention to me that I was supposed to implement this.  Except that every address record had the customer id embedded in it, so we couldn't enter an address without a customer.  \"Curtis,\" they said, \"just enter a dummy customer called 'Occupant' and attach all addresses to that.\"  Except you couldn't enter a customer without an order.  Except you couldn't enter an order without at least one item on it.  Except you couldn't enter an item unless it was listed in inventory.  Except that reserved the \"inventory\" item and made it unavailable.  Except, except, except ...  It came down to trying to create a fake customer, with a fake order, with a fake item, with a fake item category, with a \"paid\" invoice, with exceptions sprinkled throughout the codebase to handle all of these special cases and probably more that I no longer remember.  Then, and only then, could I write the code to provide \"generic\" mass mailings. Management decided it was easier to hire an outside company to handle the mailing list for them.  If they had simply had a proper database design up front, they could have reused their existing system with little trouble.  That's what bad database design costs you and why I usually start with that before writing my software.  Note: if you're not familiar with database design, here's a talk I give where I make it fairly simple to understand. I mostly avoid big words.  ","title":"Death By Database","url":"\/articles\/death-by-database.html"},{"body":"    Celebrating Ten Years  What do you do?  What makes you different from other consulting firms?  What do you bring to your clients?  Rescuing Legacy Code  Network Provisioning Automation  Data Management  Security  A\/B Testing  What are you the most proud of?  What have you learned during the past ten years?  Trust our instincts  Hiring is everything  Never give up  Why are you developing a game?  What would you like to say to your future clients?     Celebrating Ten Years  All Around the World  is now celebrating its ten-year anniversary. Managing director Leïla Contraire and I collaborated on this post to talk about what we’ve accomplished.  It has been an impressive journey working with American, European, and British clients with an international team. Stabilizing, modernizing, developing new and old code, supporting, training, managing on-site teams.  And that’s not all, we have been expanding.  While at heart we are a Perl consulting company, we have branched out into a variety of languages. We’ve done C++, Lua, TypeScript, Go, Python, Java, React, and more. We still have Perl as a core focus, but being able to diversify not only makes AAW stronger, but gives us the ability to appreciate different approaches to problems.  So after ten years we thought it would be a great time to answer the questions that people at conferences keep asking us.  What do you do?  Our job is to solve a large variety of concerns for our clients: untangling an old code base, handling the work in-house teams don’t have time for, using our expertise to develop new tools that the company has less experience with.  Old or new, there is always this excitement of being able to bring a solution. Given our many years of experience, we have seen just about everything. It doesn’t mean the work is easy—it often requires hard effort both from us and the client—it just means that no matter what is thrown at us, there is this deep certitude that we can solve it and that’s an amazing feeling.  What makes you different from other consulting firms?  In our experience when clients need a consulting company, they either want devs who are going to deliver the company strategy, or they need people who can put together a solution that will address all their concerns.  Because we care, we excel at the second. We love to come in, listen to the client concerns and aims, and propose the best solution, taking into consideration all the parameters. We love to bring flexibility in our solutions, always keeping in mind that we are not here to stay. We are just a temporary booster for a long-term adventure.  We can achieve that because our teams are special. They share our values. They are dedicated, talented, and complementary. Working with them is like being part of an orchestra. Everyone is essential, everyone is part of something bigger than themselves, and seeing them work is simply magic.  In a way to go back to your previous question, we are bringing magic to the tech world.  What do you bring to your clients?  What we bring depends on what the client needs.  Rescuing Legacy Code  Sometimes it’s untangling a mess of legacy code, transforming an old system into something maintainable. After understanding the code, we design an appropriate architecture and gradually, layer-by-layer, transform the code to that architecture. Sometimes we finish that work. Other times we evolve the code to the point where the client is capable of taking over.  Network Provisioning Automation  Of course, we also build new tools. In one case, a transatlantic telecom client was struggling to manage the costs of network provisioning due to it being a manual process. Many large projects were stalled due to time spent managing day-to-day tasks. We built a system that automated the provisioning, including security, freeing their network engineers to work on the difficult problems that actually required human intervention.  Data Management  Data management is another key area. We have found that even the best software developers often do not understand how to design a database or structure large amounts of data to be useful to their company. This often leads to complex, slow code that tries to work around the limitations of their data. We get asked to sort this out. This is often a delicate process, requiring extensive testing as we slowly restructure the data and processes to deliver the quality the client requires.  Security  Security and reliability are uppermost in everyone’s mind today, but they’re often viewed in terms of cost instead of profit. As a result, they often get ignored until the company’s hand is forced.  For one of our largest clients, most of their internal employees had full access to everything on their system. In redesigning their access control they found it both fragile and difficult to implement on a multi-million line codebase. We were brought in and over the course of two years, we deleted large swaths of their codebase, replacing it with a single configuration file and a handful of supporting modules. The project was a great success and made it trivial to have fine-grained control over access, even down to individual components on a page.  A\/B Testing  Today, much of the software world has agreed that testing their software’s behavior is important. But what about testing their customer’s behavior? Does your new sign-up form increase or decrease the number of sign-ups? Can you prove it? One of our customers could, but the A\/B testing system they built was difficult to work with. Because they put all of the data into a single database table, the system required custom SQL be written for every A\/B test. The SQL had to be run from the command line to produce reports.  We rebuilt their entire system with a properly designed database, along with beautiful front-end charts showing the performance of all tests for whatever confidence level desired, including the ability to set the minimum number of required visits per test. Creating a new test only took a few few clicks. Stopping and restarting tests, including adjusting the number of buckets was trivial. A\/B testing is a fantastic tool for discovering if your changes have the impact you’re looking for. You can stop guessing and start using a data-driven approach.  What are you the most proud of?  The certitude that, as a team, we can address any problem that is given to us.  What have you learned during the past ten years?  There are three key lessons we have learned.  Trust our instincts  If something is not right for us, it’s likely not right for the client or our teams. We know when to say “no.”  Hiring is everything  It is not just about assessing technical abilities, but also evaluating communication skills, critical thinking capacity, and the candidate’s desire to contribute to something greater. At our company, we take immense pride in a job well done.  Never give up  Even in the face of adversity, we believe there is always an opportunity for something more significant, happier, and worth building, just around the corner.  With these insights, we are better equipped to steer our company towards sustained growth and success.  Why are you developing a game?  Because we needed to create something we believe in, that belongs to us, to our teams, something that lasts. We are immensely proud of our work on Tau Station  .  As for lasting, it has 8 years of development, and we are only approaching the marketing phase.  It has been a labor of love, the sheer joy of using our experience to tell meaningful stories, and the real struggle to make it happen.  Creating something—giving life to something—is such a different adventure than consulting. It gives us perspective, understanding for our clients, as even in our code base, we sometimes see the pain of legacy code.  What would you like to say to your future clients?  Come and talk to us  . No matter the problem, the need, we have the solution.  ","title":"All Around the World: Celebrating Ten Years","url":"\/articles\/all-around-the-world-celebrating-ten-years.html"},{"body":"   Disclaimer  Social Networking is Broken  The Opportunity  The Background  The Problem  Trust  Horror  Broadcasting  The Unholy Trifecta: Trust, Horror, and Broadcasting  The Contenders  Friendster  MySpace  Orkut  hi5  Google Buzz  Yahoo! 360&deg;  Bebo  Google+  Honorable Mentions  Creating A New Social Network  Excellence  Trust, Horror, and Broadcasting  The David to Facebook's Goliath  Building Trust  Open Source  Open Business  No \"real names\" Policy  Community Engagement  Eliminating Horror  Controlled Broadcasting  AI Categorization Pipeline  Profit?  Can It Happen?     Disclaimer  Yes, I know, this is a big post. You're probably not going to read all of it and that's OK. I understand. However, if you know of others who feel that Facebook needs competition, please share this with them. You can't go up against a multi-billion dollar corporation with a tweet or a 3 paragraph blog entry.  Social Networking is Broken    Via Anthory Quintano   Chris Hughes, one of the founders of Facebook, has called for it to be dismantled  . Needless to say this generated a ton of media coverage for one simple reason: social networks are broken. They’re driving us apart instead of bringing us together. You hop onto Facebook and see old friends from school spewing vile accusations about your favorite political party. Your beloved uncle is sharing another stupid meme while you sigh and search Snopes for a URL to post in response. You see close friends in relationships publicly, bitterly, \"break up\" online where everyone can read their accusations. And that's the mild stuff; the hard stuff is much, much worse.  You jump from news stories about Facebook being used to build support for genocide  to foreign propaganda campaigns to manipulate elections  to livestreaming a mass murder in New Zealand  .  And it's not stopping, though Facebook is belatedly responding. They take down the livestreams of the regular rapes  , the gang rapes  , the child rapes  , the murders and suicides  , and more. This is somewhat better than in 2016 when Facebook refused to remove the livestream of a murder because \"it doesn&rsquo;t violate the company&rsquo;s community standards\"  .  And before we go any further, just ask yourself one question: how can a company get so wrong that they think livestreaming murders isn't violating \"community standards\"?  The Opportunity  Let's see if we can figure this out. Facebook has 2.3 billion monthly active users  , millions of whom want anything else, and 15 million of their most valuable customers have walked away since 2017  . Despite this, Facebook still basked in almost seven billion dollars in net income for the final quarter of 2018  . You would think that investors would be drooling at this. Millions of Facebook high-value customers are begging for a new market! Even more interesingly, there's little barrier to entry into this market.   Taking a percentage of Facebook's market share is a huge business opportunity but investors don&rsquo;t seem to be biting. Why is that? And can it be changed?    Guess which one is the 800-pound gorilla? Source   Well, as it turns out, investors were biting but they were also bitten. With seeing so many other social networks getting up, swinging, and missing, it's no wonder that investors today are wary. However, once you start diving into mess, you find that many otherwise savvy investors have fallen for tulip mania  and haven't paid attention to the real issues customers care about. Thus, the contenders fail.  The Background  Many of these would-be competitors have tried to take on Facebook, with the recent Google Plus shutdown being one of the more dramatic failures. Given the long history, we need a lot of backstory to unpack what's going on here.  First, let's distinguish between a General Purpose Social Network, GPSN, and a Special Purpose Social Network, SPSN, (though there's a lot of grey area here). Many of the millions leaving Facebook for Instagram  , are leaving a GPSN for a SPSN. Instagram is for sharing photos and videos. If you have no photos or videos to share, you may not find an Instagram account as appealing, though you may have one to follow others.   Twitter is closer to a GPSN, but it's still rather limited in its features (that, in itself, is a feature). Facebook, like Myspace, or the now-defunct Google Plus, is a GPSN. It has a wide variety of features that appeal to many people. Want to connect but don't want to share photos? Facebook. Want to connect and don't want to be limited to 280 characters? Facebook. Want to connect and regularly organize or participate in local events? Facebook. Want to have a richer reading experience about what your friends are doing? Facebook. Facebook has it all.  GPSNs are more geared towards general-purpose social networking without trying to shoehorn you into a particular mode of sharing. While many people enjoy a variety of social networks, Facebook dominates in terms of daily user engagement  , with half of its users logging on more than once a day.  The distinction between GPSN and SPSN is important because this article is only focusing on the former. The latter gets lots of traction in narrower markets and has a robust, lively community of competitors. GPSNs, however, have been reduced to \"Facebook or nothing\" as of this writing.  The Problem  Before we can really get down to the business case for a Facebook competitor, we need to understand why so many people are clamoring to leave Facebook. Once we do that, we can understand why so many David&rsquo;s have fallen before this Goliath. And once we do understand what is wrong with Facebook, the failures become almost obvious (given 20-20 hindsight).  First, let&rsquo;s be crystal-clear about what Facebook has done right: they&rsquo;ve created a generic product that has everyone with an internet connection as their target market . If you can connect to the net and you want to be social, Facebook is perfect for you. Of course, if you say \"everyone is my target market\" to an investor, that&rsquo;s a quick way to get them to leave, but given Facebook&rsquo;s 2 billion+ user accounts, we really do have a serious \"everyone\" claim here.      We are social by nature. Source   Second, given that so many people on the net want to be social, the second major thing that Facebook has done right is not being absolutely awful at what they do. That sounds ridiculous, but when you look at the clown car of wanna-be competitors, it&rsquo;s amazing how many of them actually got traction and then fell down because they weren&rsquo;t up to the job, technically.  Third, given the fact that the core Facebook product started out rather limited, they&rsquo;ve been smart enough to offer APIs to allow others to build value for Facebook  , something that allows them to offer much more to the public without having to build it themselves. Just look at all of those annoying game requests you get to begin to understand whats' happening here. Surprisingly, many competitors have ignored this.  So what has Facebook done wrong? Just about everything else.  After being relentlessly pummeled by Facebook-related horror stories, I finally said \"enough\" and started really digging in to understand what is wrong with Facebook. I've found three major problem areas that need to be solved, but they generally have the root cause of being a secretive, unaccountable company focused primarily on profits and market domination. As a businesperson, I will humbly suggest that there is nothing wrong with that. Honestly, you want to make money and destroy your competition? That's perfectly fine.  Where I draw the line is ethics . Facebook has such a huge impact on the public sector that excluding itself from accountability and acting in complete secrecy has allowed it to become a monster. Ironically, Facebook has been so horrifying, and there have been so few interesting competitors, that people are suffering \"compassion fatigue\" and when Facebook does something awful, they just ignore it or share it on Facebook. Things that would kill any other company are shrugged off for Facebook.  So I dug in, hard, trying to get to the bottom of this. I've read so many articles about Facebook and their atrocities that I fear I'm becoming numb to them, but that means I can give you a summary and spare you the worst.  Trust  This entire section could simply be about the Cambridge Analytica scandal  . Facebook essentially allowed Cambridge Analytica to mine Facebook's data to manipulate the US public opinion in the 2016 presidential election, and British public opinion on the Brexit vote  . Or there were the Facebook ads purchased by the Russian government to influence the US election.  , even though Facebook originally said there was no evidence for this and stonewalled US Congressional investigations  .  Or maybe you don't like Facebook, so you don't have an account. Don't worry, thanks to \"shadow profiles\", Facebook still probably knows who you are  and there's nothing you can do about it.  Or consider the time when Apple had to revoke a Facebook app's permission because Facebook cheated and signed the app with a certificate they weren't allowed to use.  . There's no excuse for this, but they cheated for the sole purpose of collecting more data about people, including people as young as 13.  And why not consider the incredibly poor security mistakes made when \"protecting\" your data? For example, Facebook stored millions of passwords in plain text  , one of the most basic of security blunders. Or how about discovering that half a billion user records were publicly available after they were uploaded to Amazon Web Services  . Facebook could put an end to that by not letting this information be collected by third-party applications, but that's not going to happen.  As for privacy, as long as Facebook maintains their real-name policy  , you will never have strong privacy guarantees. Facebook has tried to mitigate some of the issues with this policy, but their \"fix\" raised more problems than the policy itself  . Facebook is unlikely to address this issue because fake names cause the share price to drop  .  The privacy issue hits Facebook again and again and again. For many people, it's their number one reason for wanting to find another GPSN. Zuckerberg has flipped from privacy is no longer a \"social norm\"  , to insisting Facebook would become a \"privacy-focused communications platform\"  . Given Zuckerberg's history, he's more likely to be known as \"the boy who cried privacy.\"  Of course, given all of the scandals, why doesn't the board simply fire Zuckerberg as CEO and move in a new direction? Because they can't. Mark Zuckerberg has a majority stake in Facebook and can fire any director, at any time, without cause.  . The board can't lift a finger against Zuckerberg and he knows it. He has complete control over everything Facebook does.  Horror  The Myanmar military used Facebook posts to gain support for the genocide against Muslims in Myanmar  . White nationalists have long used Facebook to coordinate activities, but only recently has Facebook started banning those groups  . And a Kenyan vigilante group is using Facebook to murder people suspected of being in gangs  . These and many other scandals would be enough to destroy many companies, but \"scandal fatigue\" has definitely set in. However, this isn't where the worst of it is.  Facebook's livestreaming is mostly boring, innocuous, and sometimes touching. And every once in a while its mind-numbing horror will turn your stomach. It's incredibly painful to read about.  It includes suicides, parents murdering their children, parents abusing their children, gang rapes, and more. And do not forget the New Zealand mass murder that was livetsreamed.  Facebook (read: Zuckerberg) knows this. Despite repeatedly livestreaming horrific crimes, Facebook knows they are making a lot of money and Zuckerberg refuses to do anything about it.   They've been repeatedly asked to add a time-delay to livestreams, or remove them entirely until they can figure out how to stop this. However, they're serious about not delaying livestreams, even if there are brutal crimes involved, but they're taking a huge amount of heat over this. Their response?   “From now on, anyone who violates our most serious policies will be restricted from using Live for set periods of time — for example 30 days — starting on their first offense. For instance, someone who shares a link to a statement from a terrorist group with no context will now be immediately blocked from using Live for a set period of time,” Facebook VP of integrity Guy Rosen wrote.  Read that again. This is the \"Vice President of Integrity\" explaining that Facebook won't delay the livestreaming of rapes and murders.  Broadcasting  This issue seems relatively minor from a \"news\" point of view, but it turns out that it's one of Facebook's biggest problems. From a source inside Facebook who cannot be named: \"the number one reason people leave Facebook is because their mom joins.\" An independent study confirmed this.  There are several things going on, much of what deals with the difference between online and offline social interactions. Offline, social interactions are face-to-face. We know who we're dealing with and can adjust ourselves accordingly. Online, you're in what I call \"broadcast\" mode. You don't necessarily know who you're communicating with and nobody wants to tell a dirty joke to their grandmother.    Social media broadcasts whether you like it or not. Source   Google+ attempted to address the \"broadcasting\" issue by creating \"circles\" that you could put people into. If you posted, you could say \"just share with these circles\" and Grandma would never see that naughty joke you shared. But it was a chore trying to always remember which content you could share with which circles. People join social networks to be social, not to remember which switches they need to flip every time they want to be social.  And the Google+ circles, while trying to fix what you were broadcasting, did nothing to address what you were receiving. The broadcasting problem works both ways.  The Unholy Trifecta: Trust, Horror, and Broadcasting  So why do I focus on these three problems of Facebook? From my reading, they're the three central problems that Facebook cannot overcome (in this analysis, privacy falls under \"trust\").  If you're going to take on Facebook, you have to address all three of these areas.  This is one of the premier reasons why the various competitors have failed. The software landscape is littered with failed replacements for Microsoft Word because the programmers correctly realized that people only use a handful of Word's features, but they failed to notice that everyone uses a different handful.  I've talked to many people about Facebook, read many articles, and have seen that it's the same problem as replacing Microsoft Word. Many people don't care about the broadcasting issue, or the privacy issues, or livestreaming issues, and so on. Trying to fix just one issue ignores large swaths of people who don't care about your issue. In fact, most competitors seem fixated on the privacy issue when it's the broadcasting issue most people care about!  With that in mind, let's look at our contenders.  The Contenders  What follows is not an exhaustive list of contenders, but it's representative of the major contenders and their failures. The key takeaway here is that some contenders definitely could have taken on Facebook, but failed in almost comical ways.  Friendster  Friendster wasn't the first social network, but it's probably the best-known early contender.  They launched in 2002 and quickly racked up millions of followers, several competitors, and a $30 million buyout offer from Google.  The founder, Jonathan Abrams, turned down that offer and Friendster later failed, slain by MySpace. There were many reasons for this failure, but a critical issue was simply poor performance. As it grew in popularity, its home page could take almost a minute to load.  MySpace  MySpace, once the most popular Web site in the US, spectacularly imploded after Rupert Murdoch purchased them for $580 million.  . A large part of this seems to be that Murdoch's NewsCorp just didn't \"get\" the internet. You move fast and you react to what your users want; post-buyout, MySpace moved slowly and built a bunch of buggy, half-thought out features.  Seriously, who spends valuable developer time and money building a virtual karaoke machine? Meanwhile, Facebook was opening up their API so that third-party companies could build the features for Facebook, allowing Facebook to focus on their core products. So again, we see the death of a social network due, in part, to technical limitations which should never have happened.  Orkut    Orkut in all its glory! Source   Orkut could have become something. It was launched to much fanfare, including this bit from The Register  (emphasis mine):  Undetered by the feeding frenzy around the social networking bubble, and rebuffed by Friendster Inc, which it attempted to buy, Google has decided to build one better. Given Friendster's well-documented problems with coping with a large number of users, and Google's world class expertise in scalability , it ought to be more than up to the technical challenge.  Except ... here's what Marissa Mayer, then with Google, had to say about Orkut  :  Orkut launched in January of 2004, and as Mayer remembers it, it attracted “several million users in a few days“. This caused the network to slow down to a crawl, Mayer noted. And that, in turn, turned a lot of users away from it — at least in the U.S. “It’s all about speed here,” she said.  Vox has an interesting article about the mess that was Orkut  , but it basically amounts to a part-time project, written by Google employee Orkut Büyükkökten, getting co-opted as a Google product, even though it was never designed to scale. What's worse, because it was largely built with Microsoft products, Google engineers deemed it \"not scalable\" and by the time they finally got around to fixing it, Facebook had been launched and the damage had been done. So again, we see the death of a social network due, in part, to technical limitations which should never have happened.  hi5  Not much to say about hi5  . Launched in 2004, for a few years they were the second or third most popular social network in the US. speeli.com has an interesting summary of why hi5 went nowhere.  Amongst other things, you could see who viewed your profile and thus had the incredibly creepy experience of knowing who your cyber-stalkers were. However, their failure was probably due to the fierce competition with MySpace and Facebook, combined with poor earnings and a funding collapse  . By 2009, hi5 was focusing heavily on virtual goods for monetization and while Facebook focused on user acquisition around the globe.  Facebook, of course, had heavy VC funding, and the hi5 social network never quite seemed as polished. Perhaps if hi5's last funding round hadn't collapsed, things would be different today.  Google Buzz  (Note: we're ignoring \"Google Wave\" because, despite people talking about it as being the predecessor to Buzz, it was a collaboration tool and not a social network)  Included only for completeness: Google has a habit of launching various social media products, seeing little user engagement, and then dropping them again. In the case of Google Buzz  , many users didn't understand what it was for, it publicly displayed who you most chatted and emailed with  , and the class action lawsuit probably did little to help.  . It lasted only two years before Google put it out of its misery.  Yahoo! 360&deg;  Remember them? Me neither. Nobody really had any idea what it was for, but it was \"social\"  .  Bebo  Bebo really could have been something. At their height they had over 40 million users with strong engagement and sold themselves to AOL for a shocking $850 million!  AOL, rather than having a sound vision of where they were going, was trying to use the piles of money they were sitting on to buy something popular and stay relevant. Two years later AOL sold Bebo for $10 million.   Oops.  AOL, somehow, someway, got this bizarro idea that they would pay almost one billion dollars for a small social network and squeeze and squeeze and squeeze money out of the users. But that was never to be. They had a host of internal issues and reportedly never invested heavily in Bebo after the purchase. Bebo had great user engagement and a 3rd-party integration API that worked well; they actually had a chance, but there was no way AOL was going to earn back that money.  Google+  Google Wave lasted two years before Google killed it. Google Buzz lasted two years before Google killed it. Orkut hung on for about a decade before Google killed it.    Google+ Launch   And then there was Google+  , arriving at a time when many people were already getting sick of Facebook and hoping for a decent alternative. And then it launched.  It's hard to be sure exactly what Google was thinking, but one designer on the project described an epic tale of fear, politics, and siloed development.  . Vision? The entire thing was driven by fear of Facebook instead of \"what value can we bring to our customers?\"  And then there's Steve Yegge's infamous Google rant  . He meant it to be private, but didn't understand how to use Google+, so it was posted to the world by accident and lovingly copied, again and again, so that it wouldn't be lost if he was ordered to take it down. It can be summed up with one delightful paragraph:  Google+ is a prime example of our complete failure to understand platforms from the very highest levels of executive leadership (hi Larry, Sergey, Eric, Vic, howdy howdy) down to the very lowest leaf workers (hey yo). We all don't get it. The Golden Rule of platforms is that you Eat Your Own Dogfood. The Google+ platform is a pathetic afterthought. We had no API at all at launch, and last I checked, we had one measly API call. One of the team members marched in and told me about it when they launched, and I asked: \"So is it the Stalker API?\" She got all glum and said \"Yeah.\" I mean, I was joking, but no... the only API call we offer is to get someone's stream. So I guess the joke was on me.  Building a social network without an API is shooting yourself in the face. This lesson has been learned again and again and again. But not by Google.  Google+ introduced \"Circles\", allowed you to edit your posts (this competition finally pushed Facebook to add this oft-requested feature), Hangouts, online photo editing, and a few other nifty features, but the lack of an API and a generally \"clunky\" feel meant that Google+ was off to a bad start, despite picking up millions of users.  And then Google said \"hey, let's piss off Youtube fans!\" and required a Google+ account to comment on Youtube videos  . And yes, people were furious. Not only were they forced to have a Google+ account to use Google services, but the \"real name\" policy meant that many people, including those who have a genuine need to hide their identity, were locked out. Google finally gave up on these changes a couple of years later  . I can't help but wonder who at Google thought that force feeding a social network to their customers was really the fastest way to their hearts.  That being said, there were some nifty innovations led by Google+, but at the end of the day, without a clear vision, Google+ slowly faded away until Google put yet another product out of its misery.  Honorable Mentions  In the history of social networking, there are still a few other sites worth mentioning.  One of the most famous is Diaspora  , still in existence as the Diaspora Foundation  . They tried to raise $10,000 to compete against Facebook but quickly raised over $200,000 dollars  , despite not having written a single line of code. They were known as a \"Facebook killer\" long before entering that search term in Google turned up murderers. Sadly, their initial release was riddled with security flaws  , helping to stall momentum. Another serious issue is that, because it's decentralized, ISIS terrorists have started using it because it's almost impossible to censor them there.  Diaspora is asking users to inform on other users who might be terrorists in an attempt to get larger \"pods\" to take down suspicious accounts.  At the end of the day, end users didn't really find a \"decentralized social network\" interesting, or they simply didn't understand it. Diaspora was doomed.  Another interesting idea was minds.com  , the Libertarian-leaning  , crypto-currency backed social network.  Yes, you read that right. Someone thought that a crypto-currency based social network was just what the world needed.   Unsurprisingly, their traffic, never high to begin with, collapsed in the 2018 cryptocurrency crash. Any money you earned through them became instantly worthless. However, they are open source  and should get kudos for that, but it's not enough to save them.  If there is one social network I wanted to like, it was MeWe  , a \"censorship-free\" social network with none other than Tim Berners-Lee  , the inventor of the Web, as an advisor. But guess what happens to sites like Gab, Voat, and MeWe, when they claim they won't censor? They attract those being censored. I'm sure Berners-Lee isn't very happy with headlines like Sir Tim Berners-Lee’s app MeWe is used by neo-Nazis and perverts  :  Sir Tim Berners-Lee's app MeWe included neo-Nazi posts, upskirt “creep ... content, including posts from paedophiles and terrorist supporters.  I went and checked out MeWe myself and found myself in a cesspool of white-nationalists, \"WeTheSheeple\", MAGA-enthusiasts, \"Exposing Satanic World Government\" members, and some, I assume, are good people. It's definitely leaning \"alt-right\" and this Reddit posts summarizes it very well.   Creating A New Social Network  Note: If you have just skipped ahead to this section, this discussion won't make as much sense.  Chris Hughes, in the aforementioned article stating that Facebook must be dismantled, referred to undoing their acquisition of Instagram and Whatsapp. But those are social networks with very specific niches, the SPSNs mentioned earlier. There are still plenty of SPSNs out there. Instead, Facebook killed the GPSNs and, as Hughes stated, gave Zuckerberg a unilateral \"ability to monitor, organize and even censor the conversations of two billion people.\"  Excellence  What's needed isn't to destroy Facebook; that would risk exchanging the devil we know for the devil we don't. Instead, what we need is competition. The various attempts at competition I mentioned earlier failed for various reasons. The reasons that never should have occured are the cases of technical incompetence. Friendster, MySpace (after the NewsCorp acquisition), Orkut (the disastrous launch), Diaspora, and others not mentioned. However, I've been in this industry for far too long for me to suspect that management are going to get better at strategic planning,   or that software \"architects\" will learn to go by evidence instead of their gut, or software developers will suddenly learn how to write clean, extensible code.  If a major attempt is made to take on Facebook, the attack must be well-thought out and there must be no shortcuts on the path to technical excellence. The investors need to understand that building value is more important than saving money&mdash;none of this idiotic \"we can save money by hiring developers for $100 a day!\" You don't enter a mature, multi-billion dollar market by winging it.    Trust, Horror, and Broadcasting  As mentioned earlier, many companies have failed to replace Microsoft Word because they offered a limited subset of features  , ignoring that different users offer different subsets. Many of the would-be competitors did a good job of addressing some of the core issues of trust, horror, and broadcasting, but no one has attempted to address all of them. If you want a GPSN to compete with Facebook, addressing all of these issues up front is going to make life easier, but how do we do that?  Some (Diaspora and minds.com) have attempted to address the trust issue by being open source, but have failed in other ways, such as terrible security issues (Diaspora) or having this weird \"cryptocurrency social network\" thing that simply wasn't appealing to most (minds.com). Others have attempted to address trust by saying \"no censorship\" (too many to list!), and wind up being cesspits for racists, conspiracy theorists, and pedophiles.    No, we don't need to host this content. And yes, this is a real screenshot.   While I believe strongly in freedom of speech, not every platform needs to allow it. But what speech to censor? By law, some speech must be censored (for example, child pornography), and that censorship varies from country to country. And that raises the question of whether or not the new company should give in to demands from authoritarian countries.  The David to Facebook's Goliath  For the sake of having a name, let's call our theoretical company \" Ubuhle  \", the Zulu word for \"beauty\" or \"goodness\".    Many companies, including powerhouses like Google, have tried and failed to take on Facebook. What we need is something radically different, addressing all of the issues above. However, unlike Facebook's relentless drive for profits and dominance, Ubuhle would be focused on the needs of its users. If a social network can be thought of as a public service, the company should act in the interests of the public, not shareholders, or Zuckerberg. Part of that difference would be creating Ubuhle as a B Corporation. B Corporations are different.   Certified B Corporations are a new kind of business that balances purpose and profit. They are legally required to consider the impact of their decisions on their workers, customers, suppliers, community, and the environment. This is a community of leaders, driving a global movement of people using business as a force for good.  B Corporations include Ben & Jerry's, Amalgamated Bank, Kickstarter, and thousands more. You can make a profit and guard your soul at the same time.  Next, let's address the trifecta.  Building Trust  Open Source  Let's just get this out of the way right now: Ubuhle's software should be open source. There are many successful open source companies  and Ubuhle's strength would be the service and trust it provides, not just the code. When the code is open source, people can look to see if we're building shadow profiles, or inappropriately mining data, or using lax security practices.  Open Business  Next: open business. Facebook has taken in a lot of money from questionable sources, such as Cambridge Analytica and Russia advertising propaganda campaigns  targeting US voters. Ubuhle should have a page where users can search for all income sources, along with their details. Instead of shrouding everything in secrecy, let it be out in the open. The default position of Ubuhle is that people have a right to know this information.  No \"real names\" Policy  Next: use any name you want. While Ubuhle would certainly like to see real names because it's easier for people to connect, in reality, many people have excellent reasons to not use their real names. Imagine if someone is being stalked by an ex and doesn't want to be found. Imagine if someone from Iran is gay and doesn't want to be executed. Imagine if someone just doesn't think other people need to know their real name. Imagine tons of celebrities who, technically, can't use Facebook because their stage name doesn't match their real name. Heck, I'm better known as \"Ovid\"  in much of the software world, but I can't use that on Facebook. And trying to enforce a \"real name\" policy is simply fraught with error. Facebook has banned many people for actually using their real names  .  Community Engagement  When Ubuhle makes major business decisions, the decisions, and the reasons for them, should be shared with the public immediately. Comments should be allowed and Ubuhle should directly engage with its customers. There should be no shadowy cabal of management making secret decisions controlling the communications of people.    Eliminating Horror  This one is easy: no livestreaming unless we can figure out a way to make it safe. But we can go beyond that. Community engagement would be necessary to find an appropriate way to decide what content is acceptable. Certainly we don't need to show the exploitation of children. And while expressing support for neo-Nazis may be legal in most countries, Ubuhle doesn't have a requirement to host that content. But only by working with the community can we navigate what are, to be honest, very difficult waters. Free speech advocates may be horrified by suppressing holocaust denial posts, while others may be horrified if Ubuhle allows non-pornographic nudity. You can't win, but you can compromise.  Horror means different things to different people: livestreaming rapes and murders qualify, but so does promoting terrorism, racism, and other forms of hate speech. How would Ubuhle handle that? How would Ubuhle support free speech but provide a \"safe\" environment?  The answer, curiously, may lie in fixing the \"broadcasting\" problem.  Controlled Broadcasting  Nobody wants to tell their grandma a dirty joke. That's why Google+ created circles: you controlled what you shared and with whom. But always having to remember select the circles for posts kind of kills the whole \"I just want to share\" aspect of being social. Sure, when I'm sitting in front of grandma, I watch my tongue. But if I'm out with friends, I don't. On Facebook, you don't know if grandma is listening to what you're telling your friends and many people leave Facebook as a result.  We can fix this with content tagging and you decide the filtering.  A default Ubuhle filter that all new users might get is \"no nudity\", \"no violence\", and \"no heavily downvoted posts.\" Maybe you decide to go one further and remove politics and religion. Maybe someone else wants the politics and not the religion. Someone else might create a custom filter to skip all posts containing \"Game of Thrones\" and \"GoT\", hoping to avoid spoilers.  With this approach, you control what you see, not Ubuhle. While certain content would still need to be banned (child-pornography), you choose your experience. This filter could be applied to all content. With the default Ubuhle filter, assuming white-supremacist groups are heavily downvoted,   you'd never see them when scrolling through groups.  That's what you read, but what about what you write?  Imagine that I've set up three groups:   Family  School friends  Close friends   My school friends are mostly from Texas and while they're a bunch of great people, it's fair to say that we find each other's political views polarizing. I could create a custom filter just for that group, which excludes anything political I may write. Trust me, they would appreciate that.  What if someone's in more than one group? The most restrictive filters would apply, unless you explicitly tag someone.  But how do we actually tag the content? There are a number of ways to do this, but one way might be ...  AI Categorization Pipeline  For a first pass, there are quite a few open source AI solutions which can categorize text, image, and video content. New content would shoot through a pipeline of categorizers which would tag content, but not filter it (aside from anything illegal). Would it be perfect? No, you'd still get the occassional unwanted content. Would it be better than blasting you with any and all content that you may not want? Absolutely. And unlike Facebook's human moderators  , the AI-Pipeline would suffer no PTSD, would not need frequent breaks from work, would not need to go home to sleep, and would be very unlikely to consider a class action lawsuit against its employer.  As an added bonus, a quick Fermi estimate  suggests that one powerful box running non-stop could replace 100 human moderators. Even being off by an order of magnitude, that's a nice financial saving.    Profit?  So how do you make money off of this? Many of the competitors appear to have avoided ads. I think that was a huge mistake. People aren't idiots and they know that a social network needs to make money. Advertising has been around since before recorded history, the first TV ads appeared in 1941  , and Nielsen has been doing targeted advertising since the 1930s  . However, in the internet era, that targeted advertising is becoming invasive, with ads following us from site to site, to our phones back to our desktops.  And it gets worse. Though I doubt this was intentional on Facebook's part, they've been charged with violating the Fair Housing Act because whether or not you could view an ad for housing depended on your race or your skin color.  Facebook managed to settle the issue by ensuring that anyone \"placing housing, job, or credit ads will not be able to target users by age, gender, or zip code.\"  . But why does Facebook get to decide that?  Ubuhle should have advertising, with crystal-clear controls over what information you do and do not wish to share with advertisers.  You can even opt out of the advertising entirely. That's what trust is all about.  But why allow advertising at all? For the final quarter of 2018, Facebook&lsquo;s ad revenue was $16.6 billion dollars.  , with 93% of that coming from mobile advertising.    I'm unsure how they'll earn $16.6 billion in revenue selling emojis.   Meanwhile, Tim Berners-Lee's offering, MeWe, doesn't allow ads but is selling cartoon icon sets at $4.99 each. You do the math.  And let's be honest: well-done ads can be brilliant.     There are plenty of other ways to earn money, but advertising is the key. What would set Ubuhle apart is giving users total control over that advertising.  And if the user wants to use an ad blocker? That's fine. there is no need to waste time or money on an adblocking arms race.  .  Can It Happen?  Yes, it can. Frankly, we would love to do it  , but we have to be realistic. This would be a long-term, cash-intensive project. Between our client work and building Tau Station  , our free narrative RPG, we are not in a position to do this without major funding. If we don't do this, someone should. Hence, we're putting this information out there.  But as Facebook co-founder Chris Hughes made clear in this Time to Break up Facebook  article:  ...would-be competitors can’t raise the money to take on Facebook. Investors realize that if a company gets traction, Facebook will copy its innovations, shut it down or acquire it for a relatively modest sum. So despite an extended economic expansion, increasing interest in high-tech start-ups, an explosion of venture capital and growing public distaste for Facebook, no major social networking company has been founded since the fall of 2011.  Investors are afraid. None of the competition has really done anything particularly different from Facebook, aside from no advertising, no censorship, and no chance. Ubuhle would be establishing a pact with its customers: you get full control and, in exchange, we work with you to build a community of trust.  Building the business would start with targeting the valuable 12 to 35 year old demographic that&lsquo;s leaving Facebook  . It would take a serious marketing campaign, probably combined with an attempt to swing \"influencers\" over to the new GPSN.  Leveraging network effects will also be key. Ubuhle would need a strong API, preferably one that is similar to the Facebook API   to accomodate those developers already familiar with Facebook. Being able to add their apps to Ubuhle would help it grow with less effort. Widgets will also be a key part of this. Having tools on your web site such as \"Login with Ubuhle\", \"Share on Ubuhle\", \"Ubuhle profile badges\", and so on.  And, of course, there's the question of scale. There's a lot I could say on this topic, but that's turning to the technical side instead of the business side, so that's a different conversation. However, it needs to be well-planned because catching bugs in the design phase is one of the best ways to keep costs down  .  None of this is cheap, but the long-term value is what&lsquo;s imporant, not the development costs. . However, it involves a commitment of years to have the chance to offer solid competition to Facebook and no one wants to take them on. But creating competition to Facebook is important, so this is offered as a chance to get discussion going. I spent weeks digging through information to come up with all of this and at times, it was unpleasant work (the livestreaming research particularly turned my stomach). But we need this. Sooner or later, someone's going to take Facebook down. Sooner is better; even the co-founders agree.  Update : One of the engineers who worked on Google+ offers some great insight as to why it failed.   ","title":"How to Defeat Facebook","url":"\/articles\/how-to-defeat-facebook.html"},{"body":"   Building a Remote Team  Hiring  The CV  The Technical Test  The Structured Interview  People Skills  Managing a Remote Team  Supporting the Team  Communication Channels  Listening  Politics  Answer Questions Before They're Asked  Daily \"Stand Ups\"  Building the Product  Understanding the Company  The Architecture  Code Reviews  Our Values        We say what we do and do what we say.    Note: this is a long article because this is a complex topic. Assembling and managing remote teams is largely like assembling and managing local teams, but there are a few differences. Further, though hiring is pretty much the same, since most people have little formal background in how to conduct a proper hiring process, I go in-depth in how to do this in a solid fashion which will gain you the best employees. This will largely be explained via our process of consulting, but it's appropriate for internal use, too.   I was living in Paris a few years ago and I was contacted by an old friend living in Brighton, England. Despite exhaustive searching, his company couldn't find any local developers experienced with Perl, nor could they find any developers both willing to learn Perl and work on their old, creaky codebase. After a bit of discussion, it was pretty clear what the issues were and what needed to be done to fix them. I've done this enough that I'm practically genetically engineered for this role and I was contacted because of my deep experience in this area.   So when I received the news that they hired a junior programmer with no Perl experience and who failed their language-agnostic programming test, I was a bit surprised. He did, however, have a key qualification I lacked.    I do not want to work here. Source    He was willing to sit in a chair in their offices for eight hours a day. I was telecommuting. Oh, and he was probably cheaper.    The situation went about as well as you would suspect and my friend eventually left the company when it was clear things weren't getting better. Apparently their junior developer couldn't redesign a decades-old codebase in a language he didn't know.  So why would a company make such a silly mistake? Fear . Developers often want to rewrite instead of refactor simply because they don't know how to rebuild a large code base. Similarly, management often says \"no\" to remote teams because they don't know how to manage one.  And I get it. Managing a remote team requires a subtly different skillset than managing a local team, especially for managing communication. However, once you learn how to build and manage a remote team, you have your pick of of some of the best talent on the planet . Think about having a team of top experts at your fingertips to build real value for your company. This is the real strength of remote teams and this is how you manage them.  Building a Remote Team  You can have the greatest management skills on the planet, but if you have an awful team, you're going to struggle.  Hiring well is the most important thing a company can do.  Every bad hire is flushing money down the toilet. Every bad hire takes the place of the person you really wanted. Every bad hire distracts other employees as they have to pick up the slack. Every bad hire tells other employees \"management doesn't understand.\"  And so it's worth repeating: hiring well is the most important thing a company can do.  That said, we have to identify what you're looking for. What you want in a remote team is what you want in a local team: a team that can respond to your needs and cares about success. In fact, most of what I say here will apply to local teams. Our consulting firm, All Around the World  , has a complete hiring process that focuses on both hard (technical) and soft (interpersonal) skills.  Hiring    Appropriate Working Conditions Source    Our hiring process is in three phases:   CV (résumé) review  Technical test  Structured interview   The CV  Many would-be employees get upset that HR departments often use software to filter based on keywords and phrases, but after years of hiring, sometimes in recruiting for other companies, I understand why HR firms do that. We don't use such software, instead choosing to hand-review every CV. But honestly, CVs are often worthless. First, many people lie on their CVs  . In fact, as I've seen on a number of reports, this number appears to be increasing in the past few years.   Other CVs are terribly dull, but have great minds behind them.  Thus, many experienced HR people will tell you: just use the CV to screen out the obviously unqualified. You need someone with a Java+Hibernate background? Ignore CVs that don't have Java on them. You can probably ignore CVs that don't have Hibernate on them. Make sure your CV lists the core hard skills you really need and if that's not on the CV, ignore it. It's amazing how many CVs we filter this way.  Also looks for duration of their experience. If you need top talent, don't go with someone who used your core technology on a three-month side project.  And because CVs are almost useless, that's pretty much all I have to say.  The Technical Test    Don't stress! Source    After we filter the CVs, we then offer candidates a chance to take our technical test. A technical test must not be Trivial Pursuit! Yes, I happen to know what the $\" and $^V variables are for in Perl. I can never remember what the $&lt; or $&gt; variables are for. That's OK, because if trivia is what you're asking about, you'll get trivial employees. Instead, you're looking for people who can think clearly about your problem domain. For example, when we're hiring backend software developers, our technical test is a small program we ask them to write, using a CSV file we provide. It might look roughly like this:  Write web-based software that allows someone to upload the included CSV file and saves the data into an SQLite database. Assume more CSV files in a similar format will be uploaded later. Create reports that show ...  There are a few other things, but we give the candidates a week to complete the test and send us the results. To make sure the test is fair, I take it myself. The above version of the test took me about two hours to complete the first draft. Again, candidates get a week to do this and we don't send them the test until they say they have the time to complete it.  And here's our secret sauce: we tell the candidate that they should return something they consider \"production ready\", but we don't tell them what \"production ready\" means. Thus, instead of writing to meet our needs, they tend to show us what they think \"production ready\" means.  For us, we're looking for well-structured code, with tests and documentation. We're looking for a properly normalized database   . We're going to see if you noticed the data quality issues in the CSV file. We're going to see if your code is vulnerable to injection attacks. We're going to see if you've used appropriate modules. In short, we check many things.   and we use a checklist to track our findings. We usually won't fail you for missing just a couple of items, but very few developers pass this test.  And in return? The developers get a full code review, including references of where they can read more information about those areas on which they were weak. I've been amazed at how many developers we turn down who thank us for that code review. Many of them have never had one before!  The Structured Interview    Not how we conduct interviews. Source    The final part of our individual hiring process is the structured interview. Very few people make it this far and this is the part where most hiring processes break down. Most interviews are worthless.  From the article:  Countless studies show that the unstructured 30-minute interview is virtually worthless as a predictor of long-term performance by any criteria that have been examined. You have only slightly more chance of choosing the better of two employees after a half-hour interview as you would by flipping a coin.  You may as well just chuck half your CVs in the bin and say \"I don't hire unlucky people.\"  The alternative is the structured interview. Here's a choice quote from a meta-analsis of 200 articles and books spanning 80 years  :  In the 80-year history of published research on employment interviewing (dating back to Scott, 1915), few conclusions have been more widely supported than the idea that structuring the interview enhances reliability and validity.  The analysis is quite in-depth, but it becomes clear that we go from the unstructured interview&mdash;well-proven to be almost worthless&mdash;to the structured interview&mdash;a powerful hiring tool that does an excellent job of helping you select the candidates who will actually be a great fit.  They actually take a fair amount of time and effort to create, conduct, and evaluate, but here's the short version.  First, the interview is primarily about assessing soft skills which are harder to teach, though we do throw in some technical questions to cover potential gaps in the technical test, based on particular client needs. So you take stock of what you really need for a candidate to succeed. Our company, All Around The World  starts by interviewing the client to find out their work environment. Are there tight deadlines? Do you need to work directly with customers? Do you have to communicate with non-technical stakeholders? Have you worked with remote developers before? And so on ...  From there, we assemble a list of open-ended questions that help us to better understand how the candidate's background fits. For example, in a fast-paced environment, the candidate might find themselves dealing with tight deadlines, so we might ask a question like \"tell us about a time when you had a tight deadline and you were concerned you might not be able to deliver on time.\" And then you apply STARR.   S ituation  Describe the problem you faced.  T ask  Describe your role in addressing the problem.  A ction  What did you do to address the situation?  R esult  What was the end result?  R eflection  Is there anything you would have done differently?   By asking every candidate the same question in the same order, and ensuring you have covered all of the points of STARR for every question (and taking careful notes the entire time!), we get a thorough and relatively unbiased view of the candidate's abilities in relation to the client's needs.  You want to have at least two interviewers for every candidate and immediately after you're done with the interview, you score the candidate (we use 1 to 5) for how well they meet the client's needs for each question. And then you just add up the points.   It works amazing well and produces stellar candidates.  A few points about structured interviews are worth noting. First, they're not often used simply because most people have never heard of them. Second, as previously mentioned, they're a time-consuming hiring process, but it's worth taking more time up front to avoid getting a \"dud\" employee. Third, candidates sometimes find it stressful to have you drilling into each question in such extreme detail, so be aware of that.    People Skills  Please note that throughout the entire hiring process, you're looking for people with good people skills. A destructive personality wrecks teams. I don't care how good they are, if they're a jerk, they ruin things for everyone. They need to communicate easily, be friendly, challenge without arguing, and generally be pleasant to be around. Don't hire toxic employees.   Managing a Remote Team    Probably not one of our managers. Source    Now that you know how we hire individuals, how do you create a remote team? This is where really knowing what needs to be done shines. You need people with different, but complementary skills. Some people might be brilliant at front-end while others are brilliant at backend. You might have someone who can see all possible complications and design a system which can handle all of them, but have another person who's good at simplifying designs and applying the \"YAGNI\" (You Aren't Gonna Need It) principle ruthlessly. Another person might be very skilled at building tools which just make life easier for other developers (we like a devops person embedded in the teams directly). By having a wide variety of skills, you get more well-rounded solutions and higher quality work.  These are people who can put their ego aside and have the willingness to work together to provide the best solution. And because they have diverse skill sets, they learn to trust one another when someone else is addressing an area of their expertise.  Important: creating a team with diverse skills is even more important for a remote team. Just because you client says they have a DBA or a DevOps engineer doesn't mean those people will have time for a new team. They have other things to do and they may not prioritize you as much as the friends they work with and know.  Supporting the Team  The absolute number one obstacle (and objection) to remote teams is communication. Sometimes that quick email someone dashed off misses key details about a particular task that needs to be done. Sometimes the employee gets finished waiting on a task and is asking around for work to do ( but see my \"Project Management in Three Numbers\" article for a great way to work around that ). And you can't just pop your head over the cubicle wall and say \"hey, Sally, where in the code do we handle authorization?\". So your role as a remote manager is to ensure that everyone always has the information they need.  This need for extra communication and working very hard on this communication seems challenging, but we've found that this makes our teams often more effective than local teams because sometimes local teams make assumptions about how things are, when our team can't afford to do that.  Communication Channels  One of the beautiful things about remote work is that you skip the office chatter, the constant interruptions, the need to beg IT services to let you use a different editor, and so on. I often work with my terminal fullscreen so I can engage in deep work  on heavily demanding tasks. But when I need to talk to someone about something, I need to have quick and easy to use communication channels. And email isn't one of them.  The tools you'll adopt will be different depending on your needs, but IRC, Slack, Zoom, Google Hangouts, and other tools will all be part of that arsenal. Whatever you do, ensure that you have at least one visual means of communication (but don't forget accessibility requirements your team members may have). It's great to ask a quick question on Slack about which hashing algorithm you should use, but nothing beats a quick Google Hangout session for quickly discussing complicated topics.  Listening    Listen and listen hard . Source    Agile fans like the term servant leader  because it makes it clear that management is there to support the team getting things done. Managing remote teams reinforces this. When your team needs you, they need you. Yes, you're setting goals, providing direction, and often serve as the voice of the business, but when Oliver says \"I can't get QA to test my branch,\" then you follow up and make it happen because Oliver's job isn't fighting with QA. When the team says there's a problem, your job is to clearly identify what's wrong and get it sorted . This is much like the ScrumMaster role of removing any and all blocks for your team.  Sometimes management can seem distant or not paying attention—they often are juggling multiple things at once and this can be distracting. Managing remote teams cannot be like this. Especially when you have your daily \"stand up\" (scare quote explained later), you must be focused and care what your team is doing.  Politics  One issue I've seen about remote teams is that they have a particular sensitivity to office politics. This may seem counter-intuitive because your team members aren't standing around the office coffee machine, complaining that Martin isn't doing his job; they're working. The problem is that they're often anonymous to other teams.   You need to build bridges to the other teams and bridges aren't supported at only one end.    Note how the bridge is cleverly supported on both ends. Source    When different groups have different priorities and there are limited resources (as there always are), it's very easy for local teams to have the \"we were here first\" attitude. Or maybe your remote team needs a new database provisioned, but the people responsible for it don't know who you are and, more importantly, have other things they need to do.  Your job is to fix this, but it's a delicate task. Yes, you can often go to senior management and say the work you've contracted to do is stalled because other teams aren't cooperating. And, especially if you're on a critical project, senior management will sometimes respond by stepping on the foreheads of those who aren't helping you.  This exchanges a short-term problem for a long-term one.  Senior management doesn't want to step in every time you're struggling and this can leave the impression that you can't solve things for yourself.   As anonymity is a classic driver of politics—you're not human to them until you meet them—fix that and other problems will be much easier.  Get to know the other teams and managers, paying special attention to those you'll be working with. When you have stand-ups, invite representatives of other teams to your Zoom meeting. If they have remote meetings, offer to be present, explaining you'd like to learn more, or offer help if need be. When they need your support, give it to them, too. Are they struggling to figure out why their PostgreSQL database is running slowly and you happen to have a PostgreSQL guru on hand? Offer that support if you can. People don't want to help if they view it as a one-way street.  You are a helpful, friendly, rising tide that lifts all boats. Helping one another and making sure everyone knows one another helps to minimize politics quite a bit. But they'll still be there. No matter how important your work, the work you're not doing is important too. Some other manager might think you're the greatest thing since sliced bread, but she has a tight deadline and no, the bug your team identified won't get fixed by the next release.  That's a realistic position, but it gets called \"politics.\" When you let your team know the bug can't be fixed right now, you're supportive of your team, but of your client, too. If the team insists on the full story, tell them the truth, but tell it from the standpoint of the manager who denied the request. \"She would love to help, but she doesn't have the resources to fix this before the next release.\"  By doing this, you show support for everyone . You're like Tom Hanks as Captain John H. Miller in \"Saving Private Ryan\": you don't grumble or complain. You support. You build bridges to others and you shield your team from the mess. They will learn to trust you, they'll reserve their complaints for critical issues, and they'll have fewer distractions from their work.  Answer Questions Before They're Asked    \"Psst! I have a question!\" Source    This is one of the hardest skills to develop, but it goes a long way to ensuring you have a great team. Often your team will say things like \"we tried to extract the data but we can't because the API appears to be inconsistent.\" If your answer is \"I'll find out immediately what's going on\", that's great, but it means your team is blocked until you come back with more information.  A better answer is to realize they're dealing with an external resource and find out who's an expert you can call on right away. Then your answer is \"Damien knows the best about that system and he can provide you with documentation and answer your questions.\"  Curiously, non-technical managers are sometimes better at this. A technical manager might have some experience configuring SNMP for Cisco routers and in a meeting, might not realize that SNMPv2 is insecure and SNMPv3 is more secure. The manager may not know anything about the NETCONF alternative and be unaware that the suggested polling solution doesn't scale with SNMP.  Thus, a technical manager might gloss over details, thinking they understand (this is not a given, of course), while the experienced non-technical manager learns to swallow their pride and ask why there's a mixture of SNMPv2, SNMPv3, and NETCONF solutions being used. Your job is to listen and listen hard and if you must leave a meeting with unanswered questions, you have a list of them and will immediately start tracking them down. Don't make assumptions that \"they'll figure this out\" because as a manager, your job is to ensure that you have the answers and the information flows freely.  This also means not shying away from the pain of a late-night session reading through documentation and checking it against your notes to ensure that you haven't made bad assumptions. You're not providing solutions, your team is, but you are there to guide them and you can't if you're not able to address their questions immediately.  Remember: one of the greatest obstacles facing remote teams is the communication flow.  Daily \"Stand Ups\"  The daily stand-up is our biggest change from traditional management techniques, particularly agile ones.  The name \"stand-up\" comes from the idea that if you're all standing up and some team members are starting to lean, the conversation has gone on too long. Thus, stand-ups are often time-boxed to 15 minutes. This does not work as well with remote teams.  The idea of a standup is to have every team member answer three short questions:   What did I do yesterday?  What will I do today?  What blockers do I have?   We've found that the 15 minute time-box doesn't work as well with our \"complete communication\" style. The reason is simple. When everyone's in the same room and someone's explaining a technical problem at length, you often hear the ScrumMaster say \"let's take that offline\" because it's disrupting the flow of the meeting.   Except with remote teams, there's no \"offline.\" You can't turn to the person next to you and say \"hey, let's talk about that problem I had.\" Even if you do, you're not fully sharing the information because not everyone hears about it.    Working on code you don't understand. Source    And if someone identifies a blocker, you address it in the stand-up , you don't \"take it offline.\" Everyone on your team needs to know you're there for them if they're waiting on a code review, if they can't figure out what MyCompany::Catalyst::Role::RewritePaths does, or they have a test failure they can't figure out. Ask for solutions, discuss the issue, have someone volunteer to look at their code. And if you must, assign someone to review their pull request. Someone might grumble that they're being taken away for a code review, but they know their code will be reviewed in a timely fashion too.  We've found that our stand-ups tend to run about 45 minutes to an hour. Every day. That takes more time, but when you have a good team, they absorb more information. Even if they don't remember all of the details about why you changed your connection pooling software, when it's discussed more in-depth, they're more likely to remember that it was done and know who to talk to when they encounter difficulties with it.  We also use stand-ups as a time for chatting and socializing. This is not the norm, but it's easy to feel isolated when working remotely and this gives the team a chance to bond, to share stories, and laugh. All work and no play ... well, you know the rest. Being part of a team of friends, and not just co-workers, is important. When I go to conferences, our employees and contractors are often there and we take them out to dinner and share war stories. It's great to build loyalty, but more importantly, it's the right thing to do.  Note that this suggests a change in your hiring strategy. If candidates on your Google Hangout are getting fidgety towards the end of a long interview, that might be a warning sign (or maybe they have another appointment). If you mention your stand-up approach to candidates and they immediately react with \"stand-ups should not last more than fifteen minutes\", that's another warning sign that agile has become dogma to them rather than a set of guidelines. You're looking for candidates who are willing to communicate well. That means both being able to explain themselves clearly and to listen attentively.  Building the Product  Just taking a work specification and working through the bullet points is a common recipe for disaster for remote teams. Instead, you have to build value; the software just comes with the territory.  Understanding the Company  When we're brought in to a company to help with an issue and have hired our team, we work hard to understand the business of the company and the problem they're facing. And then there are other concerns which might impact our solution. Do we have to address PCI-compliance? Is the company impacted by GDPR? Does the company have restrictions on the types of software they allow? And what's the full stack they have available and are we allowed to offer alternatives, if necessary? Building a real-time reporting system with heavy writes using a MySQL database using MyISAM tables requires a substantially different architecture than Oracle 12c.  Once we have that information and can share it with the team, the hard work begins.  The Architecture  As I've written before , the later in a project that a bug is found, the more expensive it is to fix. It's easy to fix a leaky sink; it's disastrous to realize your foundation can't support the weight of your house.    The architecture of hard problems. Source    So we start with a careful evaluation of the problem, the kinds of data we're working with, and the resources we have at our disposal. Then we discuss possible solutions to the problem, with the entire team and their diverse skills contributing in their various areas of expertise. If possible, it's great to have a senior person from the company on hand who has deep experience with the codebase and company and can offer invaluable advice in driving the architecture.  If the problem is small, this might be a short meeting. If it's large, this might be a series of meetings. Regardless of the approach, the client must be able to review the architecture and offer appropriate feedback before signing off.  After hiring well, a strong architecture and review is the best thing you can do to reduce bugs and control costs.  Note that unlike code reviews, architecture reviews happen before implementation.  Code Reviews  After the architecture is approved, the implementation begins. How the work is organized depends on your management style, though I confess that I'm a fan of WildAgile and the three-number business case . Once a team member has a task, they might discuss it briefly to clarify the requirements, and when they're done working on it, they submit it for a code review.  After hiring and architecture, this is the third most important thing in assuring code quality. Even if your code works perfectly and your tests are passing, tests are not enough. Tests generally don't know when a public method should be private. Tests often can't point out that you forgot to test edge cases or that you missed a race condition. And tests certainly can't tell you that your technically correct code and documentation is nonetheless incomprehensible.  Thus, code reviews. When we do code reviews, we're checking that the code is correct, that it's understandable, that it's documented and that it has appropriate tests. More than half of all of our pull requests are sent back for more work, but that's normal for a team that is heavily focused on quality. The larger the pull request, the more likely it is to have some issues.  The combination of a thorough understanding of the problem domain, careful work on architecture, and deep code reviews means that when issues do arise in the software, or a change of requirements occurs, there are seldom systemic issues. This reduces the up-front cost of development and tremendously reduces maintenance costs.  Our Values     We say what we do and do what we say.    When working remotely, it's essential that the people you are working with share the same values. We work hard to find people who excel, are proud of their work, but can put aside their ego and focus on the client's needs.  For us, the value is to be able to help, to take the client to another level. We are putting our talent at the service of the client to support their growth. We are not there for billable hours. We are there for delivering; for supporting.  Whether you choose us or choose to develop your own remote team, following these guidelines will give you a superb team that can do things with software you never had the chance to do before.  ","title":"Building Remote Teams","url":"\/articles\/managing-a-remote-team.html"},{"body":" Summary  If you have EU customers and you're uncertain how GDPR impacts you, stop reading this and hire a lawyer. Now.  Disclaimer  This is not legal advice. If you have EU-resident customers, hire a lawyer. Now.  Background  I work for All Around The World , a consulting firm based in France. Not only have we been impacted by the General Data Protection Regulation (GDPR) law, our clients have as well. What we have seen is that there's often a lack of understanding of the consequences of GDPR and companies with EU customers face bankruptcy if they fall afoul of this legislation. The GDPR essentially forces companies to take security, privacy, and record keeping seriously. GDPR fines for non-compliance are designed to be severely punitive, up to and including driving companies into bankruptcy.  You do not need to suffer a data breach to be hit with these fines; if you regularly handle data on EU customers and you're found to not be complying with GDPR regulations, you may be sanctioned. What's worse is that local laws, such as the US CLOUD Act, are sometimes in conflict with the GDPR , meaning that you can be caught between contradictory legal requirements. You need a lawyer.  The EU worked for years on the law, finally adopting it on April 14th, 2016. It become enforceable on May 25, 2018. Companies had two years to get ready, so non-compliance is not tolerated. The Cambridge Analytica scandal has made the situation even worse. Though Facebook was fined £500,000 for misusing customer data , there's anger that this paltry amount was the maximum fine that could be levied at the time. Had GDPR been in effect at the time, Facebook could have been fined up to $1.8 billion . While \"good faith\" efforts to comply with the law will be taken into account, if the EU desires to make an example of a company (and they will), a US company taking money out of the EU will be a much more desirable target than bankrupting an EU company keeping money in the EU. You need a lawyer.  The Legal Situation  There is a strict requirement that you make a \"good faith\" effort to achieve GDPR compliance. Before anything else, you must realize that GDPR-compliance is a legal issue, not a technical one. If you simply instruct your IT staff to make it easy to report and purge personal data, you're well on your way to bankruptcy. This is not a good faith effort because it demonstrates a lack of knowledge of GDPR regulations.  If you are found to be non-compliant, there are two levels of fines :   Up to the greater of 4% of global annual turnover or €20 million for the most serious failures, such as insufficient customer consent.  Up to the greater of 2% of global annual turnover or €10 million for the other failures, such as not having a Data Protection Office (if required), failure to report breaches, or not conducting an impact assessment.   In other words, you face bankruptcy if you get this wrong. This is serious enough that many US-based Web sites are no longer available here in Europe . Complying with GDPR can be costly if your IT staff are inexperienced. Not to put too fine a point on it: the strong worldwide demand for software developers has led many companies to hire inexperienced developers. Even expert developers often build poor systems when faced with significant budget or time constraints. Companies rush build software to solve problems now with little thought to the long-term consequences of poorly-built software . GDPR isn't going to improve this situation, but it's going to severely punish companies who get this wrong.  Basic GDPR Compliance  In order to achieve GDPR compliance, you need to, at minimum, do the following, keeping in mind that there are major caveats to each of these:   Request consent from your customers to track their data.  Appoint a Data Protection Officer, if required.  Create a Data Protection Impact Assessment (DPIA).  Maintain records of all personal data processing activities.  Notify EU authories of data breaches with 72 hours.  Understand the \"right to be forgotten.\"  Provide customers with information on how you use their data.   How these issues are applied will vary company to company. However, keep in mind that you generally do not have the right to deny services to customers if they fail to give consent to having their data tracked . There are exceptions to consent, such as the processing of personal data for reasons of public interest, such as in the health sector . And yes, this also applies to your EU-resident employees, not just customers.  Data Protection Officers  In short: if you handle personal information on a large scale or regularly monitor data subjects, you need a Data Protection Officer (DPO) . Keep in mind that Recital 30 of the GDPR clarifies that IP addresses are personal information. Examples of companies requiring a DPO include:   Any government body (except courts) or public authority  E-commerce companies tracking customer purchases  Hospitals and clinics  Advertising firms targeting personal data  Telephone or internet providers  Virtually any company offering financial or insurance services  Geo-location services used for statistical purposes  Recruiters who store candidate information  Security firms monitoring public spaces   Small scale tracking of personal data, such as a individual doctors or lawyers, generally do not need a DPO.  Of course, the above list is not exhaustive. If you're not sure, consult a lawyer.  If you require a DPO, the following conditions apply :   They must not have a conflict of interesting (controllers, IT directors, CEOs, etc.)  They must have sufficient budget and personnel to perform their task  They report directly to top management and must not have a supervisor  They may be an outside consultant or firm, but they must not have a short or fixed-term contract  Their term must be between two to five years, but may be extended up to ten years  They must be knowledgeable in both GDPR-compliance and your internal systems (it's understood that the latter will take time)  To protect against retaliatory dismissal, they can only be dismissed with the consent of the European Data Protection Supervisor (EDPS)  They must have full power to investigate and correct GDPR compliance within the organization  They must notify the EDPS of any data processing activity that is likely to present significant risk \"to the rights and freedoms of data subjects\".   The above list, of course, is not exhaustive.  IT Considerations  Now, and only now, do we begin to touch on the IT considerations. The above was merely to give you a sense of the scale of the GDPR directives. This section is intentionally left short because it should not be viewed as a checklist to becoming GDPR-compliant. If in doubt, consult a lawyer.  The legal implications must be addressed first. One you have a DPO (if needed), and have drafted a comprehensive plan to protect your customer's data, you can start work on how to implement this. Implementing GDPR requirements without fully understanding them risks wasting money and time developing systems that are not fit for purpose. Once you have a strong understanding, however, you can begin to address the following:   Asking for consent in a clear, intelligible manner that gives consumer full control over how their data is managed  Develop reporting to track usage of all personal data  Respond to consumer requests for your usage of their personal information  Respond to consumer requests for the \"right to be forgotten\"  Ensure that disaster recovery and data backups do not restore \"forgotten\" information  Restrict internal access to sensitive data  Hire a reputable, external company to do a security audit and have developers fix discovered issues   The security assessment is critical: all the \"good faith\" in the world isn't going to protect you if you have a Experian-style data breach . The EU wants to show that the GDPR has real teeth and you don't want to be the example.  ","title":"GDPR and Bankruptcy","url":"\/articles\/gdpr-and-bankruptcy.html"},{"body":" This article is longer than most. That's because it's more important than most. If you avoid these software mistakes when starting a new project, you will save yourself a lot of grief. And by grief, I mean \"money.\"  My new client was playing the long game. They needed data for analysis. Lots and lots of data. They had been saving customer data for two years, waiting for the critical mass at which they could have their data science team dive into it. I wasn't on this project, but I was doing some work on their system when I happened to notice some warnings from their database that I was all too familiar with: Data truncated for column ... .  No, that wasn't an error; it was a warning. Many of you are familiar with that horror story: MySQL wasn't being run in STRICT or TRADITIONAL mode. . Further investigation revealed that the data being truncated was the key the client was using to link this valuable customer data together. It was impossible to reconstruct. Two years of valuable data had to be thrown away because the client didn't know how to configure MySQL.  This, and many other disasters, can be avoided by taking a little time up front to understand common errors on large projects.  The Database  I find it interesting that when I go into a shop running Oracle or MySQL, the database is usually a mess and there's lot of work to fix there. When I go into a shop running PostgreSQL, the database is usually fine. I can't prove this, but I suspect it's because people who really understand databases reach for PostgreSQL due to its obsession with data quality, while people who just use databases reach for MySQL because it's \"easy\" or Oracle because \"the company said so.\"  As a further data point, when I hire people, there's a small technical test. I've given that test hundreds of times. It was three years before a single developer returned that test with a properly normalized database. There were no \"gotchas\" and the database itself only required five tables to be normalized correctly. Many of these were excellent developers with a decade or more of experience but nonetheless turned in sloppy databases. It was trivial to demonstrate potential data anomalies in their database . Most software developers don't understand databases.  By running a default configuration of MySQL, you can get invalid dates, truncated data, NULL values for division by zero instead of an error, and so on. MySQL and MariaDB have other issues, but this is the biggest one. Every one of my clients who has used MySQL and tried to get themselves into STRICT mode have found their software assumes sloppiness in the data and can't run with a properly configured database.  Recommendation: if you must use MySQL, ensure you're using one of the recommended modes and have a competent DBA who understands database design and performance. Otherwise, if there's not strong requirement for choice of database, use PostgreSQL. I've heard DBAs complain that it takes more work to administer, but that's because motorcyles require more maintenance than bicycles, but they'll get you there faster.  Database Migrations  When dealing with new clients:  Me: \"How do you update the database?\" Client: \"Run this script. It searches through the SQL directory to find the latest SQL we checked in. It runs that in your database.\" Me: \"How do you back out the change if there's a problem?\" Client: \"Uh, there's hardly ever a problem, but we fix it by hand.\"  And things go downhill from there. I frequently meet clients with insane database migration strategies. The \"dump SQL in a directory\" for people to apply is an annoyingly common strategy. There's no clear way to roll it back and, if you're using MySQL or Oracle, if it contains DDL changes, those aren't transaction safe, so they really should be in their own migration, but they're not.  A few clients have an in-house database migration strategy involving numbered migrations. It often looks like this:  ... 213-up-add-index-on-address-state.sql 213-down-add-index-on-address-state.sql 214-up-add-customer-notes-table.sql 214-down-add-customer-notes-table.sql 215-up-add-sales-tax-sproc.sql 215-down-add-sales-tax-sproc.sql  That, at least, can allow devs to back out changes (but tricky if your DDL isn't transaction-safe), but it's amazing when you get to migration 215 and you have 30 developers, five of them need to make a datababase change an they're arguing over who gets number 216. Yes, I've seen this happen more than once.  With a naïve numbering strategy, you can't declare dependencies, you get numbering conflicts, you really can't \"tag\" a particular migration for deployment, and so on.  Or there are the migration strategies which require migrations to be written in a particular programming language. Those are often nice, but can't always leverage the strength of the database, often write very poor SQL, and make it hard for other teams not using the language to write migrations.  Recommendation : just use sqitch . It's well-tested, not tied to a particular framework or programming language, let's you write changes in native SQL, declare dependencies, tag changes, and so on.  Oh, and it's free and open source.  Time Zones  This one still surprises me, but many clients are storing datetimes in their database in \"local time.\" Most of the time this means \"not UTC.\" I'll just get the recommendation out of the way now.  Recommendation : whenever possible, store dates and times in the database in UTC.  Many shops started out small and many datetime tools report the local time because if you're in Portland, Oregon, you get confused when it's 1:30 PM but your time function says 8:30 or 9:30 PM.  But it's worse than confusion. I've seen many test suites fail twice a year ... right when DST starts and ends. That costs money!  Or here are some fun problems documented in the Perl DateTime module: Ambiguous and invalid times.  Ambiguous Local Times  Because of Daylight Saving Time, it is possible to specify a local time that is ambiguous. For example, in the US in 2003, the transition from to saving to standard time occurred on October 26, at 02:00:00 local time. The local clock changed from 01:59:59 (saving time) to 01:00:00 (standard time). This means that the hour from 01:00:00 through 01:59:59 actually occurs twice, though the UTC time continues to move forward.   Invalid Local Times  Another problem introduced by Daylight Saving Time is that certain local times just do not exist. For example, in the US in 2003, the transition from standard to saving time occurred on April 6, at the change to 2:00:00 local time. The local clock changes from 01:59:59 (standard time) to 03:00:00 (saving time). This means that there is no 02:00:00 through 02:59:59 on April 6!  Using UTC doesn't make all of your problems go away, but because UTC doesn't use DST, you'll never have invalid or ambiguous times.  Time zone issues seem rather abstract, but as your company grows, these problems become more severe. In particular, if you work with other companies, they will be very unhappy if you cut off their services an hour or two early simply because you got your time zones wrong. Or in one memorable case (NDA, sorry), a company printed up a bunch of schedules with all times an hour off because of this.  One of my clients in California tried to switch to UTC because they had gone worldwide and local time was causing them headaches. When they realized how much time and money it was going to cost, they switched to Mountain Standard Time (Arizona) because they don't use daylight savings time. My client simply accepted that all of their old dates would forever be an hour or two off.  NoSQL  One of my favorite interactions with a developer was in the Paris office of a client with legitimate \"big data\" needs. But their NoSQL solution (which I'm not going to name) was awful. It was slow, painful to query, but offered \"eventual constistency.\" There was a different NoSQL solution that seemed like a better fit, including a much better query syntax. However, if you learn one thing as a consultant, let it be this: never assume your client is stupid. If you're wrong, your client gets angry. If you're right, your client gets angry.  I approached a senior dev who was there when the NoSQL decision was made, laid out my findings, and asked \"what am I missing? I need help understanding your NoSQL choice.\"  He replied, \"we misread the documentation.\"  I laughed; he didn't. He wasn't joking. This multi-million dollar business made a very expensive mistake that they couldn't easily back out of because they misread the documentation.  This is not unusual.  Many developers, me amongst them, get excited about new technologies. Either we want to explore new things (that's why we became developers!) or we want to pad our CVs. Neither of these is necessarily in the company's best interest. While I sometimes recommend NoSQL (I love Redis, for example), usually I urge caution. I've seen client after client reach for NoSQL without understanding it. Here's what you need to understand:  Before SQL, everything was NoSQL. That's why we have SQL.  If you're considering a NoSQL solution, here's a handy checklist for you:   What problem are you trying to solve and do you really need to solve it?  Why does your database not solve that? (NoSQL is a premature optimization if you don't have a database yet)  How can you fix your database to solve that problem? (I'm amazed at how often this step is skipped)  If your current developers can't answer the previous question, have you considered hiring an expert? (Often far less expensive than introducing a new technology)  Why does a NoSQL solution solve this problem better?  What type of NoSQL solution do you really need? Key\/value? Column-oriented? Graph? And so on ...  What limitations does your NoSQL solution have and how are you compensating for these?   Often when I see clients switching from SQL to NoSQL, I find they're trading a known set of problems for an unknown set of problems. How is that good for your business? NoSQL has its place, but given how many developers are don't understand databases, why do we magically assume they're going to understand the byzantine NoSQL solutions available?  Internationalization  If there is any chance that your software will ever need support for multiple languages, investigate building that support in from day one. One of my clients decide to expand from the US to the UK. Easy, right? Hmm, date formats are different. Currency symbol is different. And the subtle spelling differences were \"labour\"-intensive to fix. It cost them so much money relative to their meager earnings abroad that they canceled the project. They might revisit it someday, but not now.  When looking at internationalization (i18n-getting your code ready for localization) and localization (l10n-the actual translation and other changes necessary), even if you're not going to launch in another language, make a \"secret\" translation feature. Heck, make it Pig Latin if you want. Even if you get it wrong, it won't really matter because it's not for public consumption. Instead, render everything in Pig Latin and check the software (better, write a spider that can find untranslated strings). One you get over the initial hurdle, it will become second nature to not hard-code strings or date formats everywhere.  Recommendation: Start your i18n work from day one. Trying to change it later is very expensive.  The Takeaway  You may have noticed a pattern in the above. Early mistakes are often hard to notice until they turn into expensive mistakes.    Source: IBM System Science Institute Relative Cost of Fixing Defects   The sooner you catch bugs, the less money they cost to fix. When you're dealing with well-known categories of bugs, you have little excuse for them. For open source databases, I can't imagine starting a new project with MySQL instead of PostgreSQL. Of course you need to store times in UTC. Just say \"no\" to NoSQL until you can prove you need it.  Of the problems I mention above, them can all be fixed in the design phase but are often caught in the maintenance phase where it's literally two orders of magnitude more expensive to fix: every dollar you didn't spend up front becomes $100 dollars later, and that's not counting the money you lost because you didn't fix it.  I'm a huge fan of agile development , but agile shouldn't mean checking your brain at the door and it definitely shouldn't mean \"don't think of the big picture.\" You'll save yourself a lot of money and grief if you take a little more time when you start a new project.  If you have any other \"up front\" design issues like those above, mention them in the comments below! Or better yet, tell me how many your project has made!  ","title":"Avoid Common Software Project Mistakes","url":"\/articles\/avoid-common-software-project-mistakes.html"},{"body":"  For those curious about the current status of the Corinna OOP project for Perl  , Paul Evans is working on it in the feature-class branch  . Today I decided to see how much of my Cache::LRU example I could get working.  Under the full proposal, Cache::LRU looks like this (it’s unclear if the :handles modifier will be implemented for the MVP):  class Cache::LRU { use Hash::Ordered; field $cache :handles(exists) { Hash::Ordered-&gt;new }; field $max_size :param :reader { 20 }; method set ( $key, $value ) { if ( $cache-&gt;exists($key) ) { $cache-&gt;delete($key); } elsif ( $cache-&gt;keys &gt;= $max_size ) { $cache-&gt;shift; } $cache-&gt;set( $key, $value ); # add to front } method get ($key) { if ( $cache-&gt;exists($key) ) { my $value = $cache-&gt;get($key); $self-&gt;set( $key, $value ); # add to front return $value; } return; } }  This is what I currently have working:  class Cache::LRU { use Hash::Ordered; # Delete arguments to constructor or else they'll cause a fatal # error when new() is called. When :param is added, this will # no longer be needed. field $max_size; ADJUST { $max_size = delete($_[0]-&gt;{max_size}) \/\/ 20 } field $cache; ADJUST { $cache = Hash::Ordered-&gt;new }; method max_size { $max_size } method set( $key, $value ) { if ( $cache-&gt;exists($key) ) { $cache-&gt;delete($key); } elsif ( $self-&gt;num_elements &gt;= $max_size ) { $cache-&gt;shift; } $cache-&gt;set( $key, $value ); # add to front } method get($key) { if ( $cache-&gt;exists($key) ) { my $value = $cache-&gt;get($key); $cache-&gt;set( $key, $value ); # add to front return $value; } return; } method num_elements() { return scalar $cache-&gt;keys; } method exists($key) { return $cache-&gt;exists($key); } }  A few things to note:   Default initializer blocks are not yet implemented  All unprocessed arguments in the constructor are fatal (delete them)  Everything else works quite nicely   And here are the tests for it:  my $cache = Cache::LRU-&gt;new; is $cache-&gt;max_size, 20, 'Value of $max_size set by ADJUST'; is $cache-&gt;num_elements, 0, '... and our cache starts out empty'; ok !defined $cache-&gt;get('foo'), 'We should not be able to fetch values we have not defined'; ok !$cache-&gt;exists('foo'), '... and our exists() method should confirm this'; ok $cache-&gt;set( foo =&gt; 42 ), 'We should be able to set cache values'; ok $cache-&gt;exists('foo'), '... and our exists() method should show the new key exists'; is $cache-&gt;get('foo'), 42, '... and get the value back'; is $cache-&gt;num_elements, 1, '... and we should have one element in our cache'; $cache = Cache::LRU-&gt;new( max_size =&gt; 2 ); is $cache-&gt;max_size, 2, 'We should be able to set the max_size with a constructor argument'; ok !defined $cache-&gt;get('foo'), 'We should not be able to fetch values we have not defined'; ok $cache-&gt;set( foo =&gt; 42 ), 'We should be able to set cache values'; is $cache-&gt;get('foo'), 42, '... and get the value back'; ok $cache-&gt;set( bar =&gt; 'asdf' ), 'We can set a different key'; is $cache-&gt;get('bar'), 'asdf', '... and get back a different value'; is $cache-&gt;get('foo'), 42, '... and get the value back'; is $cache-&gt;num_elements, $cache-&gt;max_size, '... and we see that our cache is now full'; ok $cache-&gt;set( new_element =&gt; 3 ), 'We can set a new item in our cache'; is $cache-&gt;num_elements, $cache-&gt;max_size, '... and we see that our cache is still full'; ok !$cache-&gt;exists('foo'), '... and older elements in our cache should be ejected';  The tests revealed an “off by one” error in my original code (caches would contain up to max_size + 1 elements), along with a syntax error (both now fixed), but as it stands, I think everyone should be pleased with the progress Paul is making. Exciting times for Perl!  ","title":"Current Corinna Status","url":"\/articles\/current-corinna-status.html"},{"body":"    The Proposal  A Little Background  What is Inheritance?  A Bank Account Example  Conclusion     The Proposal  There has been some discussion of whether or not the new Perl OO model, Corinna  , should support exposing field variables to subclasses:  class Parent { field $name :inheritable; } class Child :isa(Parent) { field $name :inherited; }  There are a few benefits cited there.   If field $name doesn’t exist in the parent, the child class throws a compile-time error.  If the child can access $name directly, it’s a performance win (no method call).  By being explicit about :inheritable and :inherited , we’re not exposing data by accident.   It seems like a win, but it’s not.  A Little Background  Most OO developers today use class-based OO, but JavaScript is popular enough that prototype-based OO is getting better known. Then there’s Dr. Alan Kay, the man who coined the term “object-oriented programming” five decades ago and is considered one of the fathers of the concept. For him, OOP is actually about:   Message passing  Isolation  Extreme late binding of everything   So that’s another way of looking at OOP. And by “isolation”, Kay actually said “local retention and protection and hiding of state-process.” He would not approve of exposing field variables because it’s explicitly exposing the state, not hiding it.    Yeah, well, you know, that’s just like your opinion, man.   So clearly there are different ideas about how OOP should be implemented , but Kay is focusing on how to do it safely . He has a maths and biology background and he thought about the billions of cells in our body which die every day, but we don’t. He wants that sort of robustness in object-oriented code. To him, the only truly successful demonstration of OOP is the internet. Tons of servers crash every day, but the internet does not.  In fact, before he got to this conception of OOP, his original description of OOP left out inheritance because it was so problematic. The way I like to describe it is the Person :isa(Invoice) problem. Even if you can get that to work, it simply doesn’t make sense and until AI becomes a hell of a lot better, the software has no way of knowing if what you’re doing makes sense.  What is Inheritance?  Per Wikipedia, inheritance  is:   In object-oriented programming, inheritance is the mechanism of basing an object or class upon another object (prototype-based inheritance) or class (class-based inheritance), retaining similar implementation.   Some languages, such as Perl, allow multiple inheritance (Corinna does not). Other languages, such as Java and Ruby, only allow single inheritance, but provide you with tools (interfaces and mixins, respectively) as a substitute. In fact, inheritance is widely viewed as so problematic that Go and some Visual Basic variants don’t provide inheritance at all!  What’s worse, I’ve found that many developers kind of assume that behavioral inheritance and subtype inheritance are the same thing, but they’re not. Behavioral inheritance (which is what Perl tends to use), merely uses syntax to say “hey, I’m borrowing my parent class’s behavior.” There are no guarantees. Subtype inheritance, however, ... wait? What’s a subtype?  In the Raku language  , here are two subtypes of Int : Odd and Even :  subset Even of Int where * %% 2; subset Odd of Int where !(* %% 2); my Odd $x = 4; # we get an exception here  You can use the Even and Odd subtypes anywhere you can use an Int and your program is still guaranteed to be correct, but with the added safety of knowing that those variables will always be even or odd if they’re set.  Here’s an important concept to remember (you’ll be tested on it later!): the parent type does not know, or care, about what subtypes will be created from it. You can subtype anything in Raku (or other OO languages) and while subtypes know about their parents, the reverse is not true.  So subtype inheritance, used by languages such as Eiffel or Beta, guarantees that you can use an instance of a subclass anywhere you can use an instance of a parent class and the program still works.  Of course, that’s what the Liskov Substitution Principle  is all about.  If you’re subclassing, you probably want to enforce subtype subclasses, but neither current Perl OO (nor Corinna, to be fair), can do that easily (when\/if we get types\/type constraints in Perl, it will help).  So that gives you an idea about divergent views on inheritance and was probably boring as heck. So forget about that for now and let’s move on to an example.  A Bank Account Example  You need to create a simple, in-memory Bank::Account class. Obviously, this is a toy example, but bear with me.   You instantiate it with a customer name  Initial balance is zero  You can withdraw money, but throw an exception if the balance would go below zero  You can deposit unlimited amounts of money   Because Corinna, does not (yet) support type constraints, we’ll ignore them for now.  class Bank::Account { # Of *course* this is too simplistic. The first pedant # who points this out loses 500 internet points. use My::Exceptions qw(InsufficientFunds) field $customer :param :reader; field $balance :reader {0}; method deposit ($amount) { $balance += $amount; } method withdraw ($amount) { if ( ( $balance - $amount ) &lt; 0 ) { InsufficientFunds-&gt;throw(&quot;Naughty, naughty&quot;); } $balance -= $amount; } }  OK, so far, so good. The code works fine and everyone is pleased. Later, business requirements are updated because Mr. Moneybags has tons of money, but sometimes he wants to be overdrawn a bit. Since Mr. Moneybags is an important customer, you need to create a Bank::Account::Premium class to allow premium customers to be overdrawn.  You’re a good programmer, you know you shouldn’t rewrite code, so you just want to subclass it:  class Bank::Account::Premium :isa(Bank::Account) { sub withdraw($amount) { ... } }  But what goes in the withdraw method? You properly encapsulated your first class, so now you can’t get around that. In your Bank::Account class, you make a tiny change:  field $balance :inheritable :reader {0};  class Bank::Account::Premium :isa(Bank::Account) { field $balance :inherited; sub withdraw($amount) { $balance -= $amount; } }  And now, everything works fine and you move your code into production.  You them immediately move it back out of production because production is throwing exceptions left and right. Why? Your system knows that if $instance isa Bank::Account holds true, the balance can never be less than zero and because the code knows it can trust your Bank::Account class, it’s safe.  Bank::Account::Premium is a behavioral subclass, not a subtype subclass because the behavior of the child differs from that of the parent. It allows negative balances and the parent does not.  Oh, but maybe that doesn’t happen. I’m just being paranoid, right? Or maybe it only happens in one place and you can write one tiny little hack rather than remove this valuable functionality.  Months pass and your code is running fine and you’ve moved on to another department, when you get a frantic call to find out why everything is failing. After investigation, you discover the bug. Someone’s released a module allowing :isa type constraints and the person maintaining the Bank::Account module has realized that it’s safer to constrain the $balance :  field $balance :inheritable :reader :isa(ZeroOrPositiveNum) {0};  But your code which does allow negative balances blows up because it’s inherited this field.  Remember: the parent classes don’t know how they’re going to be subclassed and they shouldn’t care. Instead, they should do everything in their power to be as correct as possible and the :isa(ZeroOrPositiveNum) constraint is perfectly appropriate.  By exposing our internal state to our subclass, we’ve tightly coupled them because the cheap :inheritable workaround seems so easy . But again, Kay reminds us of “local retention and protection and hiding of state-process.” State and process are tightly coupled inside of a class and exposing state without the process that guarantees the state is correct leads to fragile code.  Here’s the real issue: ignoring encapsulation because it’s “easy” means we don’t have to think about our object design. Instead, we should have had something like this:  class Bank::Account :abstract { field $customer :param :reader; field $balance :reader {0}; method deposit ($amount) { $balance += $amount; } method withdraw ($amount) { $balance -= $amount; } } class Bank::Account::Regular :isa(Bank::Account) { use My::Exceptions qw(InsufficientFunds) method withdraw ($amount) { if ( ( $self-&gt;balance - $amount ) &lt; 0 ) { InsufficientFunds-&gt;throw(&quot;Naughty, naughty&quot;); } $self-&gt;next::method($amount); } } class Bank::Account::Premium :isa(Bank::Account) { # no need to override any behavior, but we could # provide more behavior as needed }  Seems like more work up front, but we have known for a long time that fixing broken code in the design stage is far cheaper than fixing it after it’s in production. But we’re so used to violating encapsulation, often without even realizing it, that we fall back on this rather than using proper design.  Yes, this example is contrived, but it’s based on my decades of OO programming. I also started off as a rubbish OO developer, having learned the syntax and not the theory. My punishment was spending years working with multiple clients, cleaning up the messes of other developers with the same background.  Conclusion  Encapsulation has been accepted best practice for OOP developers for decades. For Perl developers, it’s a contentious issue, due in part to how easy it is to apply “quick fixes” without having to revisit our design. And then, everyone quotes Larry. Larry Wall famously said  :   Perl doesn’t have an infatuation with enforced privacy. It would prefer that you stayed out of its living room because you weren’t invited, not because it has a shotgun.   That might work for disciplined developers who understand OOP. But we’re often undisciplined. We often don’t understand OOP. We’re often under time pressure and it’s easy to make mistakes then.  Larry has done great things with Perl, but this saying has outlived its usefulness. For small, quick hacks, or proofs-of-concept, that’s fine. But as systems grow, it’s not fine. There’s a saying that wise men make saying so fools can repeat them. I have long been one of those fools, but years of fixing broken OOP systems in many shops have taught me that I was wrong.  Corinna likely will allow encapsulation violation via a cumbersome meta-object protocol (MOP), but this will glaringly stand out like a sore thumb as a code smell. It will be clear that there’s a design flaw and the design should be revisited.  Not being able to easily violate encapsulation seems like a burden, but only because we’ve grown lazy and have skipped thinking about design. In reality, it will help us build more robust systems and teach us when our design is flawed. Far from being a limitation, encapsulation will be a powerful tool.  ","title":"Understanding Class Inheritance","url":"\/articles\/understanding-class-inheritance.html"},{"body":"    What’s an Object?  Generic Experts  Mutability  Object-Relational Mappers  Teaching OOP?  Conclusion     Disclaimer : I’m the lead designer of the Corinna object-oriented system going into the Perl core  . I’m probably a bit biased here.   It seems like a week can’t go by without another blog entry or video explaining why object-oriented programming (OOP) is bad  . While the content of many of those articles is bad, if you read enough of them, some common themes emerge: “mutability” and “genericity.” Both are difficult to solve, so it’s worth explaining what’s wrong here. But first, what’s an object?  What’s an Object?  Let’s take an extended quote from my book Beginning Perl  .   ÆVAR THE PERSONAL SHOPPER  You’re an awfully busy person and have little free time but plenty of disposable income, so you’ve decided to hire a personal shopper. His name is Ævar (any resemblance to reviewers of this book, living or dead, is purely coincidental) and he’s friendly, flamboyant, and most of all, cheap.  Because Ævar is new to both your city and the job, you have to tell him carefully how much money he can spend, exactly what quality of products you want, and where to buy them. You may even have to tell him which route to drive to pick up the goods and how to invoice you.  That, in essence, is procedural code and that’s what you’ve been doing up to now. You’ve been carefully telling the computer every step of the way what to do.  After a few months of explaining every little detail, Ævar gets upset and says, “þegiðu maður, ég veit alveg hvað ég er að gera” (Icelandic for “Shut up dude; I know what I’m doing”). And he does. He knows what you like and where to get it. He’s become an expert. In OO terms, you might now be doing this:  my $aevar = Shopper::Personal->new({ name => 'Ævar', budget => 100 }); $aevar->buy(@list_of_things_to_buy); my $invoice = $aevar->get_invoice;  You’re no longer telling Ævar every little step he needs to take to get your shopping done. He’s an expert, and he has all the knowledge needed to do your shopping for you and present you with the bill.   That’s all objects are. They are experts about a problem domain, but that’s actually a problem.  Generic Experts  This issue is extremely important, but it’s one that’s not touched on often enough.  OOP works best in constrained environments. For a company, if your developers really understand OOP (they often don’t), OOP is great because you can create custom experts for your domain. Your objects know how your custom authentication and authorization system work, so you can share this code with other developers in the company and they don’t have to rewrite it.  Until that one team has some custom authorization rules dealing with SOX compliance, GDPR, PCI, or a host of other things you’ve never heard of. They might ask you to make your objects more “generic” to allow them to handle their custom needs, but that starts to add complexity and, potentially bugs. If enough teams ask for this, your beautiful authorization object can become an unmaintainable god object. Systems grow. Needs grow. And software often suffers as a result.  This is fundamentally a problem with abstraction . Just because I develop a good abstraction for my use case doesn’t mean I’ve developed a good abstraction for yours.  In other words, for generic OOP, it can often be problematic because your solution may not be general enough to fit other people’s needs. For Perl, the LWP  objects to handle Web communication tend to work very well because it’s a well-constrained problem space.  In contrast, my own HTML::TokeParser::Simple  module is less generally useful because it fits a particular use case that doesn’t map well to many closely-related problem spaces, For example, when you need to map out the hierarchical structure of HTML, the stream of tokens my module produces aren’t well=suited to this. Thus, you may find yourself using a class that kinda works for what you want, but is ill-suited for what you need.  Another interesting problem is dealing with scale. For example, in a small company with maybe 20 to 30 programmers, a well-designed object can be a godsend. In a company with 200 to 300 programmers, a well-designed object can often be a source of pain because it doesn’t quite fit the needs of other teams. However, it’s often hard to make an object extensible in a way that fits the needs of others because it’s hard to plan for needs that you’re not aware of. If you’re lucky, your company might have a policy that all shared objects have a shared interface (possibly with default implementations) and different teams implement them to suit their unique needs.  If you’re unlucky, different teams all implement their own Customer class and if your code interacts with their code, there’s no guarantee of interoperability. I speak from bitterly personal experience where another team implemented a User object that had only the two methods they needed, not the many, many more that were generally available. A search through their multi-million line codebase revealed many custom-made user objects, none of which really worked like the others, but any of which could potentially leak into other’s code (this is where static typing or type constraints really shine, but that’s another article).  Objects for poorly-defined problem spaces don’t scale. If you might face this, define those problem spaces before the problem starts. Let people create their own custom objects, but with the minimum required common functionality.  Mutability  The big issue that is touched on, however, is a serious problem: mutable state. You can send an object to many different areas of your code and then some code far away from you mutates the state and suddenly you find that the DateTime object you were using has a different state from what you had tested earlier.  As a general rule, sending an object somewhere is sending a reference, not a copy. Why is this bad? You can read Ricardo Signes' horror story about a bug dealing with mutable state in objects  . When you’re building a complex system and you cannot depend on the state of your objects, you are in for a world of hurt when it bites you because it can be a nightmare figuring out what unexpectedly changed the state.  This is irrelevant in the face of immutable objects. Once you fully understand why changing an object’s state is bad, your programming becomes much more robust.  If you would like to know more about the issues with immutable objects, read my why do we want immutable objects? article.   As an aside, if you switch to a service-based architecture , you can’t send references, only copies. This makes the system more robust, but the trade=off is performance. This is why cloning is often shallow instead of deep. But , if you have immutable objects, sending a reference is irrelevant because you don’t risk mutating its state. Well, unless you’re using an ORM and something outside the program mutates that state. Speaking of which ...   Object-Relational Mappers  But then we come to the real problem: Object-Relational Mappers (ORMs). ORMs are almost by default mutable (there are immutable ORMs, but they’re not common).  my $customer = Acme::Model::Data::Customer-&gt;new($id); $customer-&gt;name($new_name); $customer-&gt;save;  This is real fun when separate parts of the code fetch the same object from the database and mutate it at the same time. Optimistic offline locking  and similar strategies are your friend here.  Otherwise, you’re passing those around, it’s trivial for something to mutate the state and cause you grief. For Tau Station  , we (I) learned this the hard way and now we often fetch data from the database and simply return a copy of the data structure. It’s faster and safer that way.  But regardless of the safety of protecting the underlying data, it still doesn’t alter the fact that not only is the state mutable, it’s stored outside the program. You can be as careful as you want, even creating read-only instances of ORM objects, but if two processes pick the same object and something changes the underlying state, you can still have incorrectly functioning software. ORMs make working with databases much easier, but there is a price to pay.  Teaching OOP?  Perhaps the biggest obstacle to effective OOP is, sadly, teaching it.  When I was learning Java in college, I still remember my first Java instructor, fresh out of college, struggling to explain the different between classes and instances. You could tell she knew the difference, but she had trouble articulating it in a way that students could understand. But that’s class-based OOP. Not all OOP systems are based around classes. Prototype-based OOP  doesn’t have classes.  My second Java instructor had us build a Swing interface to read some records from a database and display them in a form. I immediately thought MVC and built model, view, and controller classes, along with a complete set of JUnit tests. My instructor rejected my code on the grounds of “I think this code might be good, but I don’t understand it. Please submit all of it as a single class.” I also had to not submit my JUnit tests.  Admittedly, my two Java instructors constitute an anecdote and not information you can actually act on, but I’ve seen this again and again why people try to explain what objects are.   Objects are data with behavior attached  Objects are smart structs  Objects are user-defined data types  Objects are polymorphic containers  ... and so on   All of those miss the point. They’re talking about the implementation or structural details of objects and not the reason for objects. Why do we have cars? So we can drive from place to place. We can argue about the engines later.  The reason we have objects, as I noted above, is that object can be experts about a problem space. I can hand an object to a junior developer who knows nothing about the underlying complexity and that junior developer can use that underlying complexity . Pretty cool, huh?  Using objects is great, but building objects? Learning how to do this well involves a bewildering array of concepts such as SOLID  , GRASP  , the Liskov substitution principle  , the Law of Demeter  , not exposing implementation details (also known as the abstraction principle  ), and so on. The last point, I might add, is why I have objected to lvalue attribute in Corinna (unless we can also have lvalue methods). You should be able to freely convert this:  field $public_key :reader :writer;  To this:  method public_key ($optional_key = undef) { # fetch key from keystore }  And your interface remains unchanged on the outside of the class, allowing you to keep your contractual promises with the class' consumers.  But many developers have asked for this:  $object-&gt;public_key = $public_key; # field # versus this: $object-&gt;public_key($public_key); # method  For the above, we’ve tightly coupled the public interface to the private implementation just to add a little syntactic sugar.   Object-oriented design is confusing enough that my friend Bob Gregory  , coauthor of the book Architecture Patterns with Python: Enabling Test-Driven Development, Domain-Driven Design, and Event-Driven Microservices  , shared an anecdote of how he taught better design to developers. He got fed up with seeing classes with names like StartHtml (which would build the start of an HTML document), that he mandated classes be named after creatures . He wound up with classes like the AuthenticationFairy and the SessionGnome , experts in their particular domains. Weird, but it appeared to work.   Since many OOP developers are self-taught, and with many OOP implementations being hard to learn, and the bewildering array of complex and sometimes contradictory “best practices” we have to learn, OOP seems to be a more advanced concept than many developers realize.  Conclusion  I think there’s a good case to be made that OOP is not nearly as useful as it’s claimed. I have worked with far too many clients whose OOP systems are, quite frankly, broken. I’ve spent a considerable about of time and energy (and thereby client money) fixing these broken systems. For one, someone had created a system whereby every time you created a new system component, you had to create a new class holding information about that component. No specialized behavior—just information. In fact, for each component, sometimes you had to add more than one new class—again, just for storing information (almost all of it strings). I proposed a different, data-driven system which, when done, would replace about 30 (and growing) classes with two (not growing).  In contrast, great OOP systems are a joy to work on and have made my life easier. I strongly recommend them, but they don’t seem terribly common.  In the research study “Productivity Analysis of Object-Oriented Software Developed in a Commercial Environment”  (Potok, Vouk, Rindos, 1999), the researchers found that there is very little quantitative evidence that OOP improves productivity in commercial environments. On the plus side, there was also little evidence that it hinders productivity. And I know that when I hand a hideously complex authorization object to a new developer, so long as they stick to the interface, it magically works . I call that a win.  I love programming in OOP and (mostly) prefer it over procedural code (we’ll skip other paradigms for now). Further, a great many programmers seem to reach for it as a first solution rather than the last, so I’ll continue to work on the Corinna OOP system to make OOP easier for Perl, but I’m hard-pressed to combat criticisms against OOP itself. For now, just remember:   Immutability is your friend  Keep your interfaces small  Well-constrained problem spaces are better for OOP  Poorly-constrained problem spaces can be great for OOP, but they don’t scale   There’s a lot more than that, but those are great starters for decent OOP programming.  Until next time, happy hacking!  ","title":"Why is Object-Oriented Programming Bad?","url":"\/articles\/why-is-object-oriented-programming-bad.html"},{"body":"    What’s an Object in Perl?  Types and Objects  Why not Moose?  Why Corinna?     As you may know, we’re designing a new OO system for the Perl language . A couple of people keep asking why we don’t just put Moose  in the Perl core.  Please note that no one is taking away bless or Moose or anything like that. If you prefer these tools, that’s fine. After all, many people strongly objected to Moose when it first came out, echoing the arguments I hear against Corinna (except that Moose was\/is slow. Corinna is not, despite the fact that no optimization has been done on the Object::Pad prototype). It took years before Moose (and later Moo) won the hearts and minds of Perl OOP developers.  So the following explanation is not for the those dead-set against Corinna. It’s for everyone else who is curious about why Perl might adopt the Corinna proposal for native object-oriented programming (hereinafter referred to as OOP), even when the Perl 5 Porters (P5P) have rejected Moose.  What’s an Object in Perl?  If you know Perl, the answer to the above question is “nothing.” Perl knows nothing about OOP. It doesn’t know what objects are. Instead, there are a few features introduced in Perl 5  to give one a basic toolkit for emulating some OOP behaviors.   An object is a data structure that knows to which class it belongs.  A class is the same thing as a package.  A method is subroutine that expects a reference to an object (or a package name, for class methods) as the first argument.  You inherit from a class by adding its name to your namespace’s @ISA array.   Does that sound like a kludge? It is, but it’s worked well enough that developers without much experience in OOP have accepted this.  It’s actually kinda rubbish because it’s more or less an assembler language for OOP behavior, but it works. In fact, it’s modeled after Python’s original implementation of objects, but with changes to take into account that we are not, in fact, Python.  So imagine a Person class written in core Perl. You have a read\/write name and that’s all. We’ll keep it simple. This is what you used to see in OOP Perl.  package Person; use strict; use warnings; sub new { my ( $class, $name ) = @_; return bless { name =&gt; $name } =&gt; $class; # this is the instance } sub name { my $self = shift; if (@_) { $self-&gt;{name} = shift; } return $self-&gt;{name}; } 1;  So you want to subclass that and create a Person::Employee and need to keep their salary private:  package Person::Employee; use strict; use warnings; our @ISA = 'Person'; sub new { my ( $class, $name, $salary ) = @_; my $self = $class-&gt;SUPER::new($name); $self-&gt;{salary} = $salary; return $self; } sub _salary { # the _ says &quot;private, stay out&quot; my $self = shift; return $self-&gt;{salary}; } 1;  OK, kinda clumsy, but it works. However, if you want to see the salary:  use Data::Dumper; print Dumper($employee);  Admittedly, salary is often very secret and the above code is rubbish, but there’s plenty of other data that isn’t really secret, but you don’t want to expose it because it’s not part of the interface and people shouldn’t rely on it. But what’s going on with that leading underscore?  In Python, you won’t get a method for that, but if you know how the method names are mangled under the hood, you can jump through hoops and call them. For Perl, you just have that method. Developers know they shouldn’t call them, but I investigated three code bases from our various clients and found:   Codebase 1: very good Perl, lots of tests, tremendous discipline, and 72 calls to private methods outside the class (or subclass) they were defined in)  Codebase 2: very sloppy Perl, few tests, and an organizational mess, but only 59 calls to private methods (to be fair, it was a smaller codebase, but not by much)  Codebase 3: a combination of miserable and wonderful Perl, almost no tests, and the largest code base we have ever worked with. About 20,000 calls to private methods.   “Private” methods in Perl are not private. At all. You have a deadline, you have a very hit-or-miss codebase, and you put something together as fast as you can. All of a sudden, the “private” code isn’t very private any more. Work with a large group of programmers with varying levels of ability? They’re going to call those private methods, they’re going to call $customer-&gt;{name} instead of $customer-&gt;name , and do all sorts of other naughty things.  What this means is that your private methods are now part of the public interface whether you like that or not. You might think you can change them, but you do so at the risk of wide-spread carnage.  If there is one things I’ve learned in decades of programming for multiple clients, it’s this:   Relying on developers to “do the right thing” is a disaster if the wrong thing is easy to do.   Corinna is designed to scale .  Types and Objects  Before we get to the discussion of why Moose isn’t going into the core, let’s consider types (we’ll skip type theory).  Types have names, allowed values, and allowed operators.  For many programmers, when they think of types, they think about int , char , and so on. Those are the names of the types.  Types have a set of values they can contain. For example, unsigned integers can hold zero and positive values up to a certain limit (determined by the amount of memory assigned to that type). You cannot assign -2 to an unsigned integer.  You also have a set of allowed operations for those types. For example, for many programming languages, multiplying the string “foo” by a number is a fatal error, often at compile time. However, Perl allows this:  $ perl -E 'say &quot;Foo&quot; * 3' 0  Of course, you should almost always enable warnings, in which case you’ll see a warning like this:  Argument &quot;Foo&quot; isn't numeric in multiplication (*) at -e line 1.  Or if you prefer, you can make these kinds of errors fatal:  use warnings FATAL =&gt; &quot;numeric&quot;;  Types such as int , char , bool , and so on, are largely there for the computer. They usually map directly to things the CPU can understand.  But what does this have to do with objects?  Objects have names (the class), a set of values (often complex), and a set of allowed operations (methods).  These map to problem domains the programmer is concerned with. In other words, they’re complex types written to satisfy developer needs, not computer needs. They’re not just a grab-bag of miscellaneous features that have been cobbled together ... well, they currently are for Perl.  Now let’s look at Moose.  Why not Moose?  First, Moose isn’t going into core because P5P said “no.” It pulls in a ton of non-core modules that P5P has said they don’t want to maintain. That should end the argument, but it hasn’t.    Truly a majestic beast.   Some argue for Moo in core, but Moo has this in its documentation:   meta  my $meta = Foo::Bar-&gt;meta; my @methods = $meta-&gt;get_method_list;  Returns an object that will behave as if it is a Moose metaclass object for the class. If you call anything other than make_immutable on it, the object will be transparently upgraded to a genuine Moose::Meta::Class instance, loading Moose in the process if required.   So if we bundle Moo we have to bundle Moose, or break backwards-compatibility on a hugely popular module.  But there’s more ...  As stated earlier, objects in Perl are:   An object is a data structure that knows to which class it belongs.  A class is the same thing as a package.  A method is subroutine that expects a reference to an object (or a package name, for class methods) as the first argument.  You inherit from a class by adding its name to your namespace’s @ISA array.   Moose is simply a sugar layer on top of that. Native objects in Perl have no understanding of state or encapsulation. You have to figure out how to cobble that together yourself.  Here’s the person class (using only core Moose and no other helper modules):  package Person; use Moose; has name =&gt; ( is =&gt; 'rw', # it's read-write isa =&gt; 'Str', # it must be a string required =&gt; 1, # it must be passed to the constructor ); __PACKAGE__-&gt;meta-&gt;make_immutable;  Right off the bat, for this simplest of classes, it’s shorter, it’s entirely declarative, and it’s arguably more correct. For example, you can’t do $person-&gt;name(DateTime-&gt;now) as you could have with the core OOP example. You could fix that with the core example with a touch more code, but you’d have to do that with every method you wrote.  You have to change construction a bit, too. Either of the following works:  my $person = Person-&gt;new( name =&gt; 'Bob' ); my $person2 = Person-&gt;new({ name =&gt; 'Bob2' });  Why two different ways? Who knows? Live with it.  But what about the subclass?  package Person::Employee; use Moose; BEGIN { extends 'Person' } has _salary =&gt; ( is =&gt; 'ro', # read-only isa =&gt; 'Num', # must be a number init_arg =&gt; 'salary', # pass 'salary' to the constructor required =&gt; 1, # you must pass it to the constructor ); __PACKAGE__-&gt;meta-&gt;make_immutable;  And to create an instance:  my $employee = Person::Employee-&gt;new( name =&gt; 'Bob', salary =&gt; 50000, );  And we don’t have a -&gt;salary method, but we can still access -&gt;_salary . Hmm, not good. That’s because Moose manages state, but doesn’t make it easy to provide encapsulation. But at least it protects the set of allowed values.  try { $employee-&gt;name( DateTime-&gt;now ); say $employee-&gt;name; } catch ($error) { warn &quot;Naughty, naughty: $error&quot;; }  And that prints something like:  Naughty, naughty: Attribute (name) does not pass the type constraint because: Validation failed for 'Str' with value DateTime=HASH(0x7ffb518206d8) at accessor Person::name (defined at moose.pl line 11) line 10 Person::name('Person::Employee=HASH(0x7ffb318d47b8)', 'DateTime=HASH(0x7ffb518206d8)') called at moose.pl line 42  Oh, but it actually doesn’t fully protect that set of values:  try { $employee-&gt;{name} = DateTime-&gt;now; say $employee-&gt;name; } catch ($error) { warn &quot;Naughty, naughty: $error&quot;; }  Note that we’ve reached inside the object and set the value directly. The above code cheerfully prints the current date and time.  Naturally, you can set the salary the same way. You don’t want people messing with private data.  And finally, Corinna.  class Person { slot $name :reader :writer :param; }  Hmm, that looks pretty easy, but what about that ugly salary problem?  class Person::Employee :isa(Person) { slot $salary :param; }  Now, internally, everything has access to $salary , but nothing outside the class does. It’s no longer part of the public API and that’s a huge win. You literally cannot set it from outside the class.  It’s also great that $salary is just a local variable and we don’t have to keep doing method lookups to get it. With Paul Evan’s   Object::Pad  test bed for Corinna, that’s been a huge performance gain, despite little work being done on optimization.  But sadly, we don’t yet have full safety on the domain of values. There’s been some discussion, but to make that work, we need to consider types across all of the Perl language. That means variable declarations, slot declarations, and signatures. We’re not there yet, but already we have something better. This, admittedly, is the biggest downside of Corinna, but we have a more solid foundation for OOP.  Why Corinna?  To be honest, encapsulation isn’t very compelling to many Perl developers. In fact, many of the best things about OOP software isn’t compelling to Perl developers because Perl doesn’t seem to have many OOP programmers with OOP experience outside of Perl, so it’s hard for them to appreciate what is missing.  As a case in point, here was a complaint from someone on the Perl 5 Porter’s mailing list that echos complaints made elsewhere:   Rather than peeling the OOP “onion” back layer by layer, build it out from what exists now. Starting with what’s needed to augment “bless”, prototypes, and overload.pm.   The problem, at its core, is that this misunderstands the problem space and current attempts to fix this.  The following is an edit of the response written by Chris Prather which nicely sums up some of the problems.   Without trying to sound offensive, this list kinda suggests you’ve not really done any extensive thought about what an object system is and should be. Most people don’t and shouldn’t ever need to. A list of things that in my opinion would need enhancement:   Classes: Perl just gives you packages with a special @ISA variable for inheritance. Packages are just a bag of subroutines, they have no idea of state.  Attributes: bless associates a package with a data structure to provide “attributes”, except it doesn’t actually provide attributes, it just provides a place to store data and leaves you to figure out what attributes are and what that means. This also means that all instance data is public by default. While we pretend that it doesn’t because Larry told us not to play with shotguns, it hasn’t stopped a lot of people putting shotgun-like things onto CPAN (or into Perl Best Practices).  Metamodel: The way you interrogate and manipulate a package is ... not obvious. Package::Stash exists on CPAN simply to provide an API for this manipulation because it’s fraught with edge cases and weird syntax.  Methods: Perl’s concept of a method is a subroutine called in a funky way. Combined with the public nature of the data, this means you can call any method on any object ... and the only thing that can prevent this is the method itself. I’ve never seen anyone write enough validation code at the beginning of their methods to deal with what is actually possible to throw at a method.  Class composition: Design Patterns: Elements of Reusable Object-Oriented Software , published literally 4 days after Perl 5.000 says to prefer composition to inheritance. Perl’s only solution to reusable behavior is inheritance. Worse, Perl supports multiple inheritance using a default algorithm that can cause weird, non-obvious bugs.  Object Construction Protocol: Ensuring that all of the attributes are initialized properly in the correct data structure during construction is left entirely as a lemma for the programmer.  Object Destruction Protocol: See above, but because Perl has universal destruction where we can’t even guarantee the order in which things are destroyed.   The fact that Perl’s built in object system just gives you a bag of primitives and leaves you to build a robust object system for every application you write is kinda the reason things like Moose exist. Moose’s choices to solve many of these problems is the reason Corinna exists. Let’s take Classes, attributes, and methods for example (because this is the most obvious change in Corinna). Classes are supposed to be a template for creating objects with initial definitions of state and implementations of behavior. Perl’s native system only provides the second half of that.   Ultimately, by using Corinna, we can have Perl know what an object type is, not just the current hodge-podge of SVs, AVs, HVs, and so on.  Subroutines and methods will no longer be the same thing, so namespace::clean and friends become a thing of the past.  We can eventually write my $object = class { ... } and have anonymous classes.  Corinna (via the Object::Pad test bed) already gets the tremendous benefits of compile-time failures if $name is missing, instead of the run-time failure of $self-&gt;name not existing.  We can have compile-time failures of trying to call instance methods from class methods, something we’ve never had before. In fact, there are many possibilities opened up by Corinna that you will never have with Moo\/se.  That’s why Moose isn’t going into core. It also gives insight into why we’re not trying to gradually improve core OOP but are instead jumping right in: you can’t get there from here because they’re fundamentally different beasts. You can’t take a handful of discrete primitives and suddenly transform them into a holistic approach to OOP. Maybe someone could map out a way of doing that in an incremental fashion and by the year 2037, the last three Perl programmers will have a perfectly reasonable OOP system. I am tired of waiting.  I’m not saying Corinna is going to make it into core, but the prospects for it look very good right now. Corinna opens up a world of possibilities that Moo\/se can’t give us because Moo\/se is trapped within the constraints of Perl. Corinna is removing those constraints.  ","title":"Why is Perl not putting Moose in the core?","url":"\/articles\/why-is-perl-not-putting-moose-in-the-core.html"},{"body":"    Introduction  The Routine  Starting the Day  Get Dressed  Set a Schedule  Starting Work  Communicate  Be Human!  Taking Breaks  Ending the Day  Conclusion     Introduction  By now, with the COVID-19 pandemic sweeping the world, many companies are turning to remote work for the first time. We’ve been having conversations with some of them, and there’s been quite a spike in traffic for my article about managing remote teams . Read that article if you want to know how to build and run remote teams. Read this article if you want to know how to be productive when remote.     We say what we do and do what we say.    Note that the following, along with the managing remote teams teams article, is a product of years of refinement because All Around the World  has been doing this for years. Whether you’re a backend Perl hacker, a front-end UI genius, or deep in the trenches of DevOps (hint: we do all of these and more), the process is the same.  But frankly, most articles about working from home are boring and sound like they’re written by that creepy guy from HR whose permanent smile never quite reaches his eyes.  Instead, let’s look at how your day goes.  The Routine  Starting the Day    Make that bed! Source    According to the Center for Disease Control, 1 in 3 adults don’t get enough sleep.  . Not only does this impact your health, it impacts your productivity at work. So get a good night’s sleep. But how?  First, get rid of your damned snooze button!  I used to hit snooze multiple times every morning, sometimes being in a frantic rush to get to work on time because I’d have hit that snooze button five or six times. In fact, I would set my alarm clock to wake me up too early because I knew that snooze button was laying in wait for me.  Every alarm clock with a snooze button I’ve ever had has a way to turn that button off. Do it. It might sound scary at first, but when you wake up and realize that there is no snooze available, you have no choice but to get up.  I was surprised at how well this simple trick works and I’ve been using it for years.    Next, when you get up in the morning, make your bed. I doesn’t have to have perfectly crisp “hospital” corners, but it should be comfy enough that when you get back to bed, your first thought isn’t about how messy and uncomfortable the bed is. Instead, it’s just lifting the covers and sliding into bed for a good night’s sleep.  Not only are there studies showing numerous benefits associated with making your bed  (with the caveat that correlation doesn’t equal causation), but it also sets the tone for the day: I’m going to do something productive.  Oh, and if you have trouble falling asleep, read this  .  Get Dressed  Don’t work in your pajamas!  This sounds silly, but after you get out of bed, take the time to brush your teeth, shower, shave (if you do), get dressed, and so on. The power of ritual and tradition is well-known for establishing positive connections and relationships. If you start work in your pajamas, you’re breaking a personal ritual (even if it doesn’t feel like one). So get up and act like you’re preparing for your commute, even if that commute is only to your home office.  The jokes about people who work from home developing poor hygiene are based in reality. On various forums, I’ve seen plenty of people lament that they’ve gotten careless. When I first started working remotely in the early 2000s, I did the same thing and discovered it was not a recipe for success.  Set a Schedule    Photo by Jon Tyson on Unsplash Source    Did you work 9 to 5 in the office? Great! Do it from home. Setting and maintaining a regular schedules is the single most important tip I can give you. Without the discipline of a regular schedule, it’s easy to get “sloppy” in the hours we work. If you’re allowed to work from home, don’t abuse the privilege. However, keep in mind that you might be able to adjust your schedule, assuming your company is understanding (some will insist upon 9-to-5 without appreciating the benefits of flextime).  Do you like to sleep in? Go ahead! If you want to work from 11-to-7, so be it. Just make sure that it’s regular and you’re still reachable by your colleagues. If you find that the best communication for your company happens after lunch, why not optimize for that and throw away the alarm clock?  Or are you a seriously early riser? Working from 6 to 2 has many advantages. Having a few early hours where you’re not constantly getting bombarded by communication from your colleagues can be very productive. And stopping work at two in the afternoon feels positively sinful . Even if you can only do that one day a week, see if you can squeeze it in. It’s great for Fridays!  Whatever hours you keep, even if you vary them, it’s best if you have a schedule posted that you can share with your colleagues. If they’re frantically pinging you on Slack and don’t realize you’re sitting on the terrace reading a book because you’re off work, things will go bad.  Starting Work  When you’re sitting down with your mug of coffee (never tea, you heathens!) at your computer, in your “work area” that doesn’t overlap your personal area (if possible), be sure to hop on Slack, IRC, or whatever your company uses for quick chatting and greet everyone with a cheery “good morning!” This sets a positive tone and let’s everyone know you’re there.  But there’s a far more important reason for doing this. When we work remotely, it’s easy to get isolated from other people, to lose those connections. Instead of just tucking straight into work, some invisible cog in some invisible machine, you’re reaching out and talking to people and reforging social bonds that are so easy to break when you’re at home all the time.  And then you start work.  But wait! There’s a trap. Quite often that “starting work” means checking Facebook, reading the news, and so on. You do that before you start working. Set your alarm earlier if you need to.  Instead, if you find yourself having trouble starting working, find a small task and do it. An active task (not “read company email”), not a passive one. Quite often you’ll find that just actively doing one small “positive” thing for your work will jump start your motor and get you going. That, and coffee.  Communicate  This is overwhelmingly your biggest challenge. Plenty of people find they can be productive while working remote, but when you’re trying to figure out how to untangle a huge mess of code you’ve been tasked with fixing, you can’t just lean over the cubicle wall and start an in-depth conversation with Sarah about best approaches. Unfortunately, this can teach you to be too self-reliant.  Small tasks? No big deal? Big tasks? Big deal. You want this knowledge-sharing. You want to discuss different approaches. You want other people to realize that changes are coming down the pike. But getting into a personal bubble and not communicating is one of the known problems with remote work.  So make sure you communicate well and often. And choose the most effective communication tools you can find. Dr. Alistair Cockburn, in his presentation Characterizing people as non-linear 1st order components in software development    , found that communication effectiveness, from best to worst, tends to be:     Face-to-face  Video  Audio  Email  Paper   Given that “face-to-face” is off the table, video is clearly the next best option. I can’t tell you how many times in a group video call I’ve seen one person explain an idea and—sometimes it’s just a subtle shift of the eyes—another person gives off a non-verbal cue that they have reservations. It’s a non-verbal cue that you won’t have with “voice only” communications and once you get used to watching for them, you won’t want to go back to voice only.  Just to restate that: use video, use video, use video !  Got it?  For quick, transient communications, Slack, IRC, or whatever tool you use is fine. If you need something more permanent, email is good, though it’s often ignored. Learning to write effective email is a skill that many have skipped. But when you need to talk, use video.  Be Human!  While you’re on those video calls (especially for your daily-stand ups), don’t just talk about your TPS reports. You’re human. Make connections. Tell jokes. Swap war stories. Talk about your lives.  You don’t want to spend the entire meeting doing this, but you need a virtual water cooler and “slack chats” ain’t it. You want to see each other, if possible. And build those human connections. Your colleagues aren’t cogs churning out work; they’re human. If you want them to see you as you instead of as another cog, you need to give them the same respect.  Note: this is less important if you only work from home occassionally. Having the face-to-face time with colleagues makes all the difference in the world.  Taking Breaks    Take your breaks away from the computer Source    This one should be self-explanatory, but it’s not. Take your breaks. And don’t take them in front of your work computer.  Go for a walk. Move about. Do something physical instead of sitting in a chair and rapidly moving your fingers for eight hours.  And while we’re at it: don’t take lunch at your desk. You’re a human being and you need variety and physical activity. Physical activity can be as simple as walking, and there are multiple benefits to even small amounts of daily walking  .  Ending the Day  In the office, would you simply get up and walk out without saying goodbye to your colleagues? Just because we’re online doesn’t mean we ignore social norms. Let your colleagues know you’re off for the day! It’s a social activity, sets clear boundaries, and if someone is trying to reach you after, it’s more likely that one of your collegues will have noticed and let them know that you’re not there.  Conclusion  Working remotely isn’t the same as working in an office. It’s not inherently a social activity, and a five-second commute isn’t condusive to maintaining physical activity. Worse, it’s easy to slack off.  Being social, physical, and productive isn’t hard to do when you’re working from home, but you’ll need to be more mindful of it. Following the simple tips in this article will make that easier.  ","title":"Work From Home Effectively","url":"\/articles\/work-from-home-effectively.html"},{"body":"    Preface  What’s an Object?  Recommendations  Immutable Objects  Recommendation  Small Interfaces and Encapsulation  Recommendation  Type Library  Recommendation  Subclassing  Recommendation  Performance  Recommendation  Problems  Too Many Responsibilities  Recommendation  Use Methods for Attribute Data  Recommendation  Confusing Subroutines and Methods  Recommendation  Inconsistent Return Types  Recommendation  Ignored Constructor Arguments  Recommendation  Leftovers  Dr. Frankenstein  Hire Us     Preface  After spending almost three years as the lead designer of the Corinna object system for the Perl core  , I’m delighted that work on including this in the Perl language will start soon. Until it’s there, our company  still does a lot of work with clients, fixing their object-oriented code (OOP) and we’ve seen some very common causes of failures. Some say that our recommendations are too strict, and for personal projects, they might be. But we work with large-scale codebases, often written in Perl, and often containing over a million lines of code. When you’re working at that scale, especially when coordinating with many developers who may not know your code well, you need to be more rigorous.  This is not a tutorial on OOP design. That in itself is a complex topic (and perhaps one of the strongest arguments against OOP). So we’ll assume you need to fix an existing OOP codebase rather than design a new one.  Note: while these examples are in Perl, many of these problems apply to codebases we’ve worked on in other OO languages.  Warning : whenever we link to a module on the CPAN, please read the documentation of that module if you decide to use it. There are often caveats, such as with the “strict constructor” modules we mention later.  What’s an Object?  First and foremost, we must keep in mind what an object is. This is covered more in other articles , but for this one, just keep in mind that an object is an expert about a particular topic. Forget all of those other explanations about objects being “structs with behavior”, or “a reference to a data type that knows what class it belongs to.” Those are implementation details that don’t tell you objects are for . In my opinion, they even hinder people’s understanding of objects because they keep you focused on the wrong thing. Objects are experts on a particular problem space, that’s all.  Recommendations  We could talk about SOLID  , GRASP  , the Liskov substitution principle  , the Law of Demeter  , and so on, but while those are important, you can read about those on your own. Instead, we’ll cover recommendations and common problems. You’ll need to use your best judgment to decide how (or if) they apply to your codebase.   Note: if this article makes your eyes glaze over, but you love the Moose OOP system, you might skip ahead to Dr. Frankenstein to see a small module built on top of Moose that implements some of the recommendations in this article.   Immutable Objects  I’ve written before why immutable objects are good , so I won’t go into too much detail. Sometimes it’s not practical to have immutable objects (e.g., caches), but immutability should be your default position. In most languages, objects are passed by reference, not value. The more widely used an instance of your object is, if it’s mutable, the more likely that code “A” will change the state in a way that code “B” didn’t expect.  Recommendation  Even if you’re not convinced, try writing and using immutable objects. Like anything new, it can take some getting used to, but there’s a payoff there. Just don’t insist that everything be immutable. Your colleagues will hate you.  Note that if your object must return references to data, those are often mutable. However, you can declare those as immutable, too.  package Some::Class { use Const::Fast 'const'; # set up data here sub get_data_structure { my $self = shift; const my %hash =&gt; ( name =&gt; $self-&gt;name, giggity =&gt; $self-&gt;_something_else, ); return \\%hash; } }  In the above, the get_data_structure method returns an immutable data structure. Attempts to access unknown hash keys or change any of the values is a fatal error (use exists to test hash keys, of course). Some people call it overkill. Others call it safety. Your mileage may vary.  If you prefer, you can deeply clone the data structure before returning it. By presenting a copy, different consumers calling that method will always have the same results, but if the returned reference itself is shared, you could have issues. That being said, this is an issue with all code, not just OOP.  Small Interfaces and Encapsulation  The interface that your class presents is your “contract” with everyone who uses your class. You should design that interface with care, but you shouldn’t expose what you don’t need to. Using one of my “go to” examples of the Corinna OOP system that will be available in upcoming releases (but not in the immediate future), here’s a basic LRU (least recently used) cache:  class Cache::LRU { use Hash::Ordered; field $cache :handles(exists) { Hash::Ordered-&gt;new }; field $max_size :param :reader { 20 }; method set ( $key, $value ) { if ( $cache-&gt;exists($key) ) { $cache-&gt;delete($key); } elsif ( $cache-&gt;keys &gt;= $max_size ) { $cache-&gt;shift; } $cache-&gt;set( $key, $value ); # add to front } method get ($key) { if ( $cache-&gt;exists($key) ) { my $value = $cache-&gt;get($key); $self-&gt;set( $key, $value ); # add to front return $value; } return; } }  With the popular Moose OOP system for Perl, that would look something like this (skipping the sub bodies):  package Cache::LRU { use Moose; use Hash::Ordered; use namespace::autoclean; has '_cache' =&gt; ( is =&gt; 'ro', init_arg =&gt; undef, default =&gt; sub { Hash::Ordered-&gt;new }, ); has 'max_size' =&gt; ( is =&gt; 'ro', default =&gt; 20 ); sub set { ... } sub get { ... } __PACKAGE__-&gt;meta-&gt;make_immutable; }  For raw Perl, it might even look like this:  package Cache::LRU; use strict; use warnings; use Hash::Ordered; sub new { my ( $class, $max_size ) = @_; $max_size \/\/= 20; return bless { max_size =&gt; $max_size, _cache =&gt; Hash::Ordered-&gt;new, }, $class; } # more code here  For both Moose and core Perl, it’s hard to encapsulate your objects and minimize your interface. For both of those, you can call $object-&gt;{_cache} to get the underlying cache. For Moose, you can also call $object-&gt;_cache (it’s very cumbersome in Moose or Moo to not expose methods for what should be private data). This means that you have exposed that data, no matter how nicely you’ve asked people to “stay out of the kitchen.” This means that if someone is accessing your “private” data, if you want to switch your internal cache to use Redis, SQLite, or something else, you can easily break the code of others who are relying on it.  We almost always see $object-&gt;_private_method or $object-&gt;{private_data} in client codebases and that’s one of the first things we try to fix, if we have time. As a class author, you need to know you can safely change the internals of your object.  Recommendation  Keep your classes as small as you can and stop making accessors public by default. For now, this means prefixing methods with an underscore to make them “private by convention.” With Corinna, you simply don’t provide :reader or :writer attributes:  class Cache::LRU { use Hash::Ordered; field $cache { Hash::Ordered-&gt;new }; field $max_size :param :reader { 20 }; ...  In the above, you can read (but not write) the max size, but there’s no direct access possible to the $cache (this will be possible via the MOP, the meta-object protocol, but we do not want violating encapsulation to be a natural default. Hitting the MOP to violate encapsulation will stand out like a coal pile in a ballroom in code reviews).   Note: some Perl developers use “inside-out” objects to enforce encapsulation. The CPAN has Object::InsideOut  and Class::InsideOut  , amongst others. While it’s easy to use instances of these classes, they were cumbersome to write and sometimes buggy. As a result, we do not experience them often in client code.   Type Library  As systems grow, it’s very easy to have this problem:  sub get_part_by_id ($self, $id) { return $self-&gt;_inventory_schema-&gt;resultset('Part')-&gt;find($id); }  Most of the time that works, but sometimes you get no result. What happened? Maybe your primary key is a UUID but you’ve supplied an integer? Oops. So now you have to rewrite that:  sub get_part_by_id ($self, $id) { unless ( $id =~ qr\/^[0-9a-f]{8}(?:-[0-9a-f]{4}){2}-[0-9a-f]{12}$\/is ) { croak(&quot;ID ($id) does not look like a UUID.&quot;); } return $self-&gt;_inventory_schema-&gt;resultset('Part')-&gt;find($id); }  Do you really want to write that? Do you really want to debug that? (there’s a small bug in that example, if you care to look for it). If you use UUIDs frequently, you don’t want to write that again.  If you’ve used Moose, you know how easy it is to use type constraints:  has 'order_items' =&gt; ( is =&gt; 'ro', isa =&gt; 'ArrayRef[HashRef]', writer =&gt; '_set_order_items', );  Of course, if you spell the isa as ArrayRef[HasRef] , you won’t find out until runtime that you’ve misspelled it. For these and other situations, just create a type library as a centralized place to put your types and share them across your codebase as needed. If there’s too much code, focus on critical paths first.  Recommendation  Creating a type library for the first time can be daunting. Here’s a simple one, based on the excellent Type::Tiny  by Toby Inkster. It will cover most of your basic needs and you can extend it later with custom types, if needed.  package My::Personal::Types; use strict; use warnings; use Type::Library -base; use Type::Utils -all; use Type::Params; # this gets us compile and compile_named our @EXPORT_OK; BEGIN { # this gets us most of our types extends qw( Types::Standard Types::Common::Numeric Types::Common::String Types::UUID ); push @EXPORT_OK =&gt; ( 'compile', 'compile_named', ); } 1;  Using it is simple.  use My::Personal::Types qw(compile UUID); sub get_part_by_id ($self, $id) { state $check = compile(UUID); ($id) = $check-&gt;($id); return $self-&gt;_inventory_schema-&gt;resultset('Part')-&gt;find($id); }  In Moose, use these constraints to turn misspelled types into compile-time failures (and to get a much richer set of allowed types):  use My::Personal::Types qw(ArrayRef HashRef); has 'order_items' =&gt; ( is =&gt; 'ro', isa =&gt; ArrayRef [HashRef], writer =&gt; '_set_order_items', );  Despite the Type::Tiny name, the manual is quite extensive  . Go there for more information.  Subclassing  A subclass of a class is intended to be a more specialized version of that class. A Car isa Vehicle , or a Human isa Mammal . However, it’s easy to get this wrong under the pressure of deadlines, complex code bases, or just plain not paying attention. Is Item isa Product correct? Or is Product isa Item more correct?  Ultimately, the problem manifests itself in what I call the “person\/invoice problem”: Person isa Invoice . That makes absolutely no sense, but your software doesn’t know that . Your software only does what you tell it to do. It can’t evaluate semantics (at least, not yet), and if you write code that runs, but doesn’t make sense, that’s too bad.  In fact, inheritance is so problematic that some OOP languages disallow it altogether, favoring composition instead. Some only allow single inheritance, but provide alternatives (mixins, interfaces, traits, etc.). As a general rule, we recommend you use roles instead of parent classes:   Role::Tiny  (for any OO code)  Moose::Role  (for Moose)  Moo::Role  (for Moo)   There’s also my own Role::Basic  which can be used where Role::Tiny is appropriate, but the philosophy of that module is different  and presents somewhat different features.  Sometimes inheritance is the right thing to do. For example, in the Test::Class::Moose  xUnit framework, we have “test control methods” which run code before the class is instantiated, before each method is run, after each method is run, and after the test class has finished running. A test_setup method might look like this:  sub test_setup { my $test = shift; $test-&gt;next::method; $test-&gt;load_fixtures(qw\/Customer Orders\/); }  In the above example, the $test-&gt;next::method is used to call the parent test_setup to ensure that all setup is ready before you try to handle this class’s setup. In fact, you might have your test_setup call a parent test_setup which in turns calls its parent test_setup . This is common in xUnit testing and the order in which events fire is important. With roles, this is often done with method modifiers, but the order in which they fire is often dependent on their load order and that is not guaranteed. If you find yourself frequently using method modifiers in roles, you might want to think about inheritance to ensure that you have complete control over the sequencing of commands.  Recommendation  We should prefer composition or roles over inheritance. Composition is good when you clearly have an object to delegate to. Roles are good when you have some behavior which might apply to unrelated classes (such as serialization to JSON or XML).  There’s much more we could say about roles, but a full tutorial is beyond the scope of this article.  Performance  Don’t stress about performance, really. It’s not the code. It’s the database. It’s always the database. Unless it’s the network. Or it’s disk I\/O. Or, or, or ...  Steve McConnell, in his fantastic book Code Complete, 2nd edition, writes:   It’s almost impossible to identify performance bottlenecks before a program is working completely. Programmers are very bad at guessing which four percent of the code accounts for 50 percent of the execution time, and so programmers who optimize as they go will, on average, spend 96 percent of their time optimizing code that doesn’t need to be optimized. That leaves little time to optimize the four percent that really counts.   Unless you know, at design time, you’ll have performance critical code (don’t write ray-tracing software in Perl, folks!), design a great system and worry about performance only when it’s proven to be an actual problem. Devel::NYTProf  is your friend here. Be aware that benchmarking can be an arcane art.  But when we talk about performance, whose performance are we talking about? The software or the software developer? Here’s a little benchmark for you:  #!\/usr\/bin\/env perl use strict; use warnings; use Benchmark 'cmpthese'; sub use_a_loop { my @numbers; foreach my $i ( 0 .. 9 ) { $numbers[$i] = $i \/ ( $i + 1 ); } return \\@numbers; } sub direct_assignment { my @numbers; $numbers[0] = 0 \/ 1; $numbers[1] = 1 \/ 2; $numbers[2] = 2 \/ 3; $numbers[3] = 3 \/ 4; $numbers[4] = 4 \/ 5; $numbers[5] = 5 \/ 6; $numbers[6] = 6 \/ 7; $numbers[7] = 7 \/ 8; $numbers[8] = 8 \/ 9; $numbers[9] = 9 \/ 10; return \\@numbers; } cmpthese( 1_000_000, { 'use_a_loop' =&gt; \\&amp;use_a_loop, 'direct_assignment' =&gt; \\&amp;direct_assignment, } );  Do you think the loop or the direct assignment is faster? Do you really care? Well, it should be pretty clear that the loop is much easier to maintain. The direct assignment, however ...  Rate use_a_loop direct_assignment use_a_loop 970874\/s -- -50% direct_assignment 1923077\/s 98% --  Whoa! Directly assigning the data is twice as fast as the loop! If something like that is at the bottom of a tight loop and benchmarking shows that it’s a performance issue, then yes, switching from a loop to direct assignment might be an idea, but that would kill developer performance when it comes to maintaining that code. If you must do that, document it carefully, perhaps including a snippet of the code that this replaces.  Recommendation  Design the system well and don’t worry about performance while building it. Doing so runs the risk of optimizing code that doesn’t need to be optimized and possibly makes the code harder to maintain, raising long-term costs.  Instead, if your code is suffering performance issues, benchmark your code and find out where the real problems are. Here’s a video of Tim Bunce explaining how to profile your code:     Problems  The above were general recommendations for OO code, but now let’s talk about common problems we encounter with our clients.  Too Many Responsibilities  For our Project 500 contract, our client was providing online credit card services for a major, well-publicized event. However, they had a serious problem: their code could only handle 39 credit card transactions per second per server. For this event, they had a contractual obligation to improve performance by an order of magnitude . That’s right; they were required to have their code run at least ten times faster. We had two weeks to develop a proof of concept (spoiler: we succeeded).  In digging in, it turns out that they had developed an in-house OO system and an in-house ORM. We quickly discovered that a single “charge $x to the credit card” transaction generated almost 200 calls to the database! This was one of their major bottlenecks.  For our client’s ORM, every time a request was made it would gather a bunch of metadata, check permissions, make decisions based on whether or not it was using Oracle or PostgreSQL, check to see if the data was cached, and then check to see if the data was in the database. Instantiating every object was very slow, even if there was no data available. And the code was creating—and throwing away without using—hundreds of these objects per request.  We considered using a “pre-flight” check to see if the data was in the database before creating the objects, but there was so much business logic embedded in the ORM layer that this was not a practical solution given our time constrain. And we couldn’t simply fetch the data directly because, again, the ORM had too many non-ORM responsibilities built into it.  On another system, we had an immutable object (yay!) that had disk I\/O and heavy data validation every time it was instantiated. Yet that data never changed between releases, so I was tasked with caching the object data. I restructured to class to separate the validation of instance data and the setting of instance data. Then I added a few lines of code to the class to handle this and it worked like a charm, but my tests missed an edge case where some data wasn’t cached properly because one bit of data was set during validation. I had made the classic mistake of putting too much logic in that class.  To address this, I built a simple class to properly cache objects on web server restart and it cached the object for me. Not only did it work right this time, I now had a flexible in-memory cache for other objects. Further, because the cache internals were encapsulated, if we want to switch the cache out for Redis or something else, it becomes trivial.   .  Recommendation  Don’t have your objects try to do too much. The “single-responsibility” principle generally means there should only be a single reason to change a class. In the real-world, this is easy to miss, but it will save you much grief if you get this right.  Use Methods for Attribute Data  In old-school OOP code for Perl, you might see something like this:  package Customer; use strict; use warnings; use Carp 'croak'; sub new { my ( $class, $name, $title ) = @_; croak(&quot;Name required&quot;) unless defined $name; return bless { name =&gt; $name, title =&gt; $title, }, $class; } sub name { $_[0]-&gt;{name} } sub title { $_[0]-&gt;{title} } # more code here 1;  So far, so good. But the business rules state that if the customer has a title, they must always be referred to by both their title and name, never just one or the other. Developers keep forgetting (or don’t know) this business rule, but remember: your object is supposed to be the expert here and it won’t get it wrong, so the object should handle this responsibility.  Now you’re rewritten the class to remover the title accessor and to provide a name method to encapsulate this logic:  package Customer; use strict; use warnings; use Carp 'croak'; sub new { my ( $class, $name, $title ) = @_; croak(&quot;Name required&quot;) unless defined $name; return bless { name =&gt; $name, title =&gt; $title, }, $class; } # always prefix their name with a title if it exists sub name { my $self = shift; return $self-&gt;{title} ? &quot;$self-&gt;{title} $self-&gt;{name}&quot; : $self-&gt;{name}; } # more code here 1;  Again, this code looks correct, but by eliminating the accessor for our title attribute, if we were forced to subclass this, how do we override it, especially if it’s used elsewhere in the class? We could just overwrite the name and title values in the blessed hash of the subclass and that might be good enough, but if we need to convert an attribute to a method—as we did with name we can’t easily do that now.  Recommendation  This is one of the advantages of Moose and Moo. You automatically get method accessors for data attributes, so users of those OO systems will rarely notice this unless they also get in the bad habit of doing $self-&gt;{title} .  Otherwise, if you’re writing core Perl, just include simple accessors\/mutators for your data (prefacing them with a leading underscore if they should be private):  sub title ($self) { return $self-&gt;{title}; } sub set_title ($self, $title) { $self-&gt;{title} = $title; }  Whether you prefer to overload a method to be both an accessor and a mutator, or to return the invocant on mutation are arguments that are far beyond the scope of this article, so we would recommend following the current style of your codebase. If it doesn’t exist, define it and stick to it (predictable interfaces are fantastic!).  Confusing Subroutines and Methods  Your Moose\/Moo classes should generally resemble the following:  package Some::Class { use Moose; use namespace::autoclean; # class definition here # make_immutable is a significant performance boost __PACKAGE__-&gt;meta-&gt;make_immutable; }  I was writing some client software that could cache some objects (it’s a coincidence that I’ve had to do this so often) and for one edge case, it was good if we could clone the objects first. Many things cannot be easily cloned, so if the object provides its own clone method, we used that instead of simply returning the object from the cache. It looked sort of like this:  sub fetch ( $class, %arg_for ) { my $object = $class-&gt;_fetch_for_identifier( $arg_for{identifier} ); if ($object) { return $object-&gt;can('clone') ? $object-&gt;clone : $object; } else { return $class-&gt;_instantiate_and_cache($arg_for); } }  That failed miserably because some objects were importing a clone subroutine and because Perl doesn’t have a distinction because subroutines and methods (though it will when Corinna is implemented), clone was in the namespace and the can('clone') method returned true. We tried to call a subroutine as a method, even though it wasn’t.  Recommendation  Using namespace::autoclean  or namespace::clean  will remove the imported subroutines from the namespace and make it much harder for code to call subroutines as methods. Read the documentation for each to decide which will work better for you.  Note that as of this writing, there is an outstanding RFC for Perl to allow a lexical importing mechanism which should make this less of a problem in the future if it’s adopted.  Inconsistent Return Types  This isn’t just an OOP problem, but it’s a common issue we see in code in dynamic languages, so it’s worth mentioning here. If you’re going to return an array reference or a hash reference, you usually want to ensure that’s all you return. For example:  my $results = $object-&gt;get_results; if ($results) { foreach my $result ($result-&gt;@*) { # do something } }  But what if you forget the if ?  my $results = $object-&gt;get_results; foreach my $result ($result-&gt;@*) { # do something }  If get_results returns undef when there are no results, the code blows up. However, if it returns an empty array reference, the loop is simply skipped. You don’t have to remember to check the return value type (unless returning nothing is an actual error). This is easy to fix in methods, but be wary of a similar trap with attributes:  has 'results' =&gt; ( is =&gt; 'ro', isa =&gt; Maybe[ ArrayRef ], );  Recommendation  There are several ways to handle this, including defining coerced types, but one of the simplest and safest:  has 'results' =&gt; ( is =&gt; 'ro', isa =&gt; ArrayRef, default =&gt; sub { [] }, );  In the above, we drop the Maybe and default to an array reference if no argument is supplied to the constructor. With that, if they pass an arrayref of hashrefs for results , it works. If they don’t pass results at all, it still works.  However, if they pass anything else for results , including undef , it fails. That’s great safety because while you want the types you return to be predictable, it’s also good to allow the types you pass in to be predictable. There are definitely exceptions to this, but they should be used with care.  Ignored Constructor Arguments  Going back to our results example:  package Some::Class; use Moose; has 'results' =&gt; ( is =&gt; 'ro', isa =&gt; ArrayRef, default =&gt; sub { [] }, );  What happens if we do my $object = Some::Class-&gt;new( result =&gt; $result ); .  No matter what the type of $result is, the value is thrown away because by default, Moose and Moo simply discard unknown arguments to the constructor. If you misspell a constructor argument, this can be a frustrating source of errors.  Recommendation  Fortunately, the fix for this is simple, too: MooseX::StrictConstructor  . (Or MooX::StrictConstructor  for Moo):  package Some::Class; use Moose; use MooseX::StrictConstructor; has 'results' =&gt; ( is =&gt; 'ro', isa =&gt; ArrayRef, default =&gt; sub { [] }, );  Now, passing unknown arguments is a fatal error.  If you’re using traditional bless syntax, you’ll have to manually validate the arguments to the constructor yourself, but the type library we outlined above can be used.  Leftovers  There’s a huge amount more we can say, including useful design patters, being cautious with method modifiers in roles, when to use abstract classes instead of roles, but we’re starting to get into the long tail of code smells in object-oriented code bases, so perhaps those are good for another article.  Dr. Frankenstein  If you’ve gotten this far, congrats! One thing very helpful with OO code is to develop a set of guidelines on how you’ll build OO code and stick to it. We’re going to be Dr. Frankenstein and build our own object system out of the parts we have laying around. By ensuring everyone uses this object system, we can have greater consistency in our code.  Let’s pretend you’ve settled on the Moose OOP system as the basis of your own. You’d like to ensure several things are true and you’ve come up with the following list:   Unknown arguments to the constructor are fatal  It should be easy to see which attributes are or are not required in the constructor  Attributes should default to read-only  namespace::autoclean must always be used  You want signatures and types  The Carp module’s carp , croak , and confess functions should always be present  You want the C3 MRO (though you should avoid multiple inheritance)  Your work uses v5.22, so you’ll enforce those features   For this, you’ve decided that param should replace has if the parameter is required in the constructor, and field should replace has if the parameter is not allowed in the constructor. Let’s use Moose::Exporter  to set this up.  We won’t explain how all of this works, so you have some homework to do.   Again, we’re not saying the above is what you should do; we’re just giving an example of what you can do.   package My::Personal::Moose; use v5.22.0; use Moose (); use MooseX::StrictConstructor (); use Moose::Exporter; use mro (); use feature (); use namespace::autoclean (); use Import::Into; use Carp qw\/carp croak confess\/; Moose::Exporter-&gt;setup_import_methods( with_meta =&gt; [ 'field', 'param' ], as_is =&gt; [ \\&amp;carp, \\&amp;croak, \\&amp;confess ], also =&gt; ['Moose'], ); sub init_meta { my ( $class, @args ) = @_; my %params = @args; my $for_class = $params{for_class}; Moose-&gt;init_meta(@args); MooseX::StrictConstructor-&gt;import( { into =&gt; $for_class } ); warnings-&gt;unimport('experimental::signatures'); feature-&gt;import(qw\/signatures :5.22\/); namespace::autoclean-&gt;import::into($for_class); # If we never use multiple inheritance, this should not be needed. mro::set_mro( scalar caller(), 'c3' ); } sub field { my ( $meta, $name, %opts ) = @_; # default to read-only $opts{is} \/\/= 'ro'; # &quot;has [@attributes]&quot; versus &quot;has $attribute&quot; foreach my $attr ( 'ARRAY' eq ref $name ? @$name : $name ) { my %options = %opts; # copy each time to avoid overwriting # forbid setting `field` in the constructor $options{init_arg} = undef; $meta-&gt;add_attribute( $attr, %options ); } } sub param { my ( $meta, $name, %opts ) = @_; # default to read-only $opts{is} \/\/= 'ro'; # it's required unless they tell us otherwise $opts{required} \/\/= 1; # &quot;has [@attributes]&quot; versus &quot;has $attribute&quot; foreach my $attr ( 'ARRAY' eq ref $name ? @$name : $name ) { my %options = %opts; # copy each time to avoid overwriting if ( exists $options{init_arg} &amp;&amp; !defined $options{init_arg} ) { croak(&quot;You may not set init_arg to undef for 'param'&quot;); } $meta-&gt;add_attribute( $attr, %options ); } } 1;  With that, and the My::Personal::Types above, here’s a (silly) example of how to use this:  #!\/usr\/bin\/env perl use lib 'lib'; use Test::Most; package My::Names { use My::Personal::Moose; use My::Personal::Types qw( compile Num NonEmptyStr Str PositiveInt ArrayRef ); use List::Util 'sum'; # removed my namespace::autoclean param _name =&gt; ( isa =&gt; NonEmptyStr, init_arg =&gt; 'name' ); param title =&gt; ( isa =&gt; Str, required =&gt; 0 ); field created =&gt; ( isa =&gt; PositiveInt, default =&gt; sub { time } ); sub name ($self) { my $title = $self-&gt;title; my $name = $self-&gt;_name; return $title ? &quot;$title $name&quot; : $name; } sub add ( $self, $args ) { state $check = compile( ArrayRef [Num] ); ($args) = $check-&gt;($args); carp(&quot;no numbers supplied to add()&quot;) unless $args-&gt;@*; return sum( $args-&gt;@* ); } __PACKAGE__-&gt;meta-&gt;make_immutable; } my $person = My::Names-&gt;new( name =&gt; 'Ovid', ); is $person-&gt;name, 'Ovid', 'name should be correct'; ok !defined $person-&gt;title, '... and no title'; cmp_ok $person-&gt;created, '&gt;', 0, '... and a sane default for created'; ok !$person-&gt;can('sum'), 'subroutines have been removed from the namespace'; is $person-&gt;add( [qw\/1 3 5 6\/] ), 15, 'Our add() method should work'; throws_ok { My::Names-&gt;new( name =&gt; 'Ovid', created =&gt; 1 ) } 'Moose::Exception', 'Attributes not defined as `param` are illegal in the constructor'; my $doctor = My::Names-&gt;new( name =&gt; 'Smith', title =&gt; 'Dr.' ); is $doctor-&gt;name, 'Dr. Smith', 'Titles should show up correctly'; cmp_ok $doctor-&gt;created, '&gt;=', $person-&gt;created, '... and their created date should be correct'; done_testing;  Obviously, the above code probably won’t be fit for purpose, but it shows you the basics of how you can build an OO system to fit your company’s needs, rather than allowing everyone to just “do their own thing.”  Hire Us  We do code reviews, development, testing, and design, focused on reliability and scalability. Even if you’re just exploring possibilities, feel free to contact us and let’s see what we can do to make your software better.  ","title":"Common Problems in Object-Oriented Code","url":"\/articles\/common-problems-in-object-oriented-code.html"},{"body":"  Originally located at http:\/\/www.pphsg.org\/cdsmith\/types.html , this article explained some basic concepts of type systems. Unfortunately, that page is gone and I had to fetch it from the web archive of that page  . The note at the bottom states that contents are in the public domain. This was published in 2008.  What follows is a short, brilliant introduction to the basic concepts of type systems, by Chris Smith.   What To Know Before Debating Type Systems  I would be willing to place a bet that most computer programmers have, on multiple occasions, expressed an opinion about the desirability of certain kinds of type systems in programming languages. Contrary to popular conception, that’s a great thing! Programmers who care about their tools are the same programmers who care about their work, so I hope the debate rages on.  There are a few common misconceptions, though, that confuse these discussions. This article runs through those I’ve encountered that obscure the most important parts of the debate. My goal is to build on a shared understanding of some of the basic issues, and help people get to the interesting parts more quickly.  Classifying Type Systems  Type systems are commonly classified by several words, of which the most common are “static,” “dynamic,” “strong,” and “weak.” In this section, I address the more common kinds of classification. Some are useful, and some are not.  Strong and Weak Typing  Probably the most common way type systems are classified is “strong” or “weak.” This is unfortunate, since these words have nearly no meaning at all. It is, to a limited extent, possible to compare two languages with very similar type systems, and designate one as having the strong* er * of those two systems. Beyond that, the words mean nothing at all.  Therefore: I give the following general definitions for strong and weak typing, at least when used as absolutes:   Strong typing: A type system that I like and feel comfortable with  Weak typing: A type system that worries me, or makes me feel uncomfortable   What about when the phrase is used in a more limited sense? Then strong typing, depending on the speaker or author, may mean anything on the spectrum from “static” to “sound,” both of which are defined below.  Static and Dynamic Types  This is very nearly the only common classification of type systems that has real meaning. As a matter of fact, its significance is frequently under-estimated. I realize that may sound ridiculous; but this theme will recur throughout this article. Dynamic and static type systems are two completely different things, whose goals happen to partially overlap.  A static type system is a mechanism by which a compiler examines source code and assigns labels (called “types”) to pieces of the syntax, and then uses them to infer something about the program’s behavior. A dynamic type system is a mechanism by which a compiler generates code to keep track of the sort of data (coincidentally, also called its “type”) used by the program. The use of the same word “type” in each of these two systems is, of course, not really entirely coincidental; yet it is best understood as having a sort of weak historical significance. Great confusion results from trying to find a world view in which “type” really means the same thing in both systems. It doesn’t. The better way to approach the issue is to recognize that:   Much of the time, programmers are trying to solve the same problem with static and dynamic types.  Nevertheless, static types are not limited to problems solved by dynamic types.  Nor are dynamic types limited to problems that can be solved with static types.  At their core, these two techniques are not the same thing at all.   Observing the second of these four simple facts is a popular pastime in some circles. Consider this set of presentation notes  , with a rather complicated “the type system found my infinite loop” comment. From a theoretical perspective, preventing infinite loops is in a very deep sense the most basic possible thing you can do with static types! The simply-typed lambda calculus, on which all other type systems are based, proves that programs terminate in a finite amount of time. Indeed, the more interesting question is how to usefully extend the type system to be able to describe programs that don’t terminate! Finding infinite loops, though, is not in the class of things most people associate with “types,” so it’s surprising. It is, indeed, provably impossible with dynamic types (that’s called the halting problem; you’ve probably heard of it!). But it’s nothing special for static types. Why? Because they are an entirely different thing from dynamic types.  The dichotomy between static and dynamic types is somewhat misleading. Most languages, even when they claim to be dynamically typed, have some static typing features. As far as I’m aware, all languages have some dynamic typing features. However, most languages can be characterized as choosing one or the other. Why? Because of the first of the four facts listed above: many of the problems solved by these features overlap, so building in strong versions of both provides little benefit, and significant cost.  Other Distinctions  There are many other ways to classify type systems. These are less common, but here are some of the more interesting ones:   Sound types. A sound type system is one that provides some kind of guarantee. It is a well-defined concept relating to static type systems, and has proof techniques and all those bells and whistles. Many modern type systems are sound; but older languages like C often do not have sound type systems by design; their type systems are just designed to give warnings for common errors. The concept of a sound type system can be imperfectly generalized to dynamic type systems as well, but the exact definition there may vary with usage.  Explicit\/Implicit Types. When these terms are used, they refer to the extent to which a compiler will reason about the static types of parts of a program. All programming languages have some form of reasoning about types. Some have more than others. ML and Haskell have implicit types, in that no (or very few, depending on the language and extensions in use) type declarations are needed. Java and Ada have very explicit types, and one is constantly declaring the types of things. All of the above have (relatively, compared to C and C++, for example) strong static type systems.  The Lambda Cube. Various distinctions between static type systems are summarized with an abstraction called the “lambda cube.” Its definition is beyond the scope of this article, but it basically looks at whether the system provides certain features: parametric types, dependent types, or type operators. Look here  for more information.  Structural\/Nominal Types. This distinction is generally applied to static types with subtyping. Structural typing means a type is assumed whenever it is possible to validly assume it. For example, a record with fields called x, y, and z might be automatically considered a subtype of one with fields x and y. With nominal typing, there would be no such assumed relationship unless it were declared somewhere.  Duck Typing. This is a word that’s become popular recently. It refers to the dynamic type analogue of structural typing. It means that rather than checking a tag to see whether a value has the correct general type to be used in some way, the runtime system merely checks that it supports all of the operations performed on it. Those operations may be implemented differently by different types.   This is but a small sample, but this section is too long already.  Fallacies About Static and Dynamic Types  Many programmers approach the question of whether they prefer static or dynamic types by comparing some languages they know that use both techniques. This is a reasonable approach to most questions of preference. The problem, in this case, is that most programmers have limited experience, and haven’t tried a lot of languages. For context, here, six or seven doesn’t count as “a lot.” On top of that, it requires more than a cursory glance to really see the benefit of these two very different styles of programming. Two interesting consequences of this are:   Many programmers have used very poor statically typed languages.  Many programmers have used dynamically typed languages very poorly.   This section, then, brings up some of the consequences of this limited experience: things many people assume about static or dynamic typing that just ain’t so.  Fallacy: Static types imply type declarations  The thing most obvious about the type systems of Java, C, C++, Pascal, and many other widely-used “industry” languages is not that they are statically typed, but that they are explicitly typed. In other words, they require lots of type declarations. (In the world of less explicitly typed languages, where these declarations are optional, they are often called “type annotations” instead. You may find me using that word.) This gets on a lot of people’s nerves, and programmers often turn away from statically typed languages for this reason.  This has nothing to do with static types. The first statically typed languages were explicitly typed by necessity. However, type inference algorithms - techniques for looking at source code with no type declarations at all, and deciding what the types of its variables are - have existed for many years now. The ML language, which uses it, is among the older languages around today. Haskell, which improves on it, is now about 15 years old. Even C# is now adopting the idea, which will raise a lot of eyebrows (and undoubtedly give rise to claims of its being “weakly typed” -- see definition above). If one does not like type declarations, one is better off describing that accurately as not liking explicit types, rather than static types.  (This is not to say that type declarations are always bad; but in my experience, there are few situations in which I’ve wished to see them required. Type inference is generally a big win.)  Fallacy: Dynamically typed languages are weakly typed  The statement made at the beginning of this thread was that many programmers have used dynamically typed languages poorly. In particular, a lot of programmers coming from C often treat dynamically typed languages in a manner similar to what made sense for C prior to ANSI function prototypes. Specifically, this means adding lots of comments, long variable names, and so forth to obsessively track the “type” information of variables and functions.  Doing this prevents a programmer from realizing the benefits of dynamic typing. It’s like buying a new car, but refusing to drive any faster than a bicycle. The car is horrible; you can’t get up the mountain trails, and it requires gasoline on top of everything else. Indeed, a car is a pretty lousy excuse for a bicycle! Similarly, dynamically typed languages are pretty lousy excuses for statically typed languages.  The trick is to compare dynamically typed languages when used in ways that fit in with their design and goals. Dynamically typed languages have all sorts of mechanisms to fail immediately and clearly if there is a runtime error, with diagnostics that show you exactly how it happened. If you program with the same level of paranoia appropriate to C - where a simple bug may cause a day of debugging - you will find that it’s tough, and you won’t be actually using your tools.  (As a side comment, and certainly a more controversial one, the converse is equally true; it doesn’t make sense to do the same kinds of exhaustive unit testing in Haskell as you’d do in Ruby or Smalltalk. It’s a waste of time. It’s interesting to note that the whole TDD movement comes from people who are working in dynamically typed languages... I’m not saying that unit testing is a bad idea with static types; only that it’s entirely appropriate to scale it back a little.)  Fallacy: Static types imply upfront design or waterfall methods  Some statically typed languages are also designed to enforce someone’s idea of a good development process. Specifically, they often require or encourage that you specify the whole interface to something in one place, and then go write the code. This can be annoying if one is writing code that evolves over time or trying out ideas. It sometimes means changing things in several different places in order to make one tweak. The worst form of this I’m aware of (though done mainly for pragmatic reasons rather than ideological ones) is C and C++ header files. Pascal has similar aims, and requires that all variables for a procedure or function be declared in one section at the top. Though few other languages enforce this separation in quite the same way or make it so hard to avoid, many do encourage it.  It is absolutely true that these language restrictions can get in the way of software development practices that are rapidly gaining acceptance, including agile methodologies. It’s also true that they have nothing to do with static typing. There is nothing in the core ideas of static type systems that has anything to do with separating interface from implementation, declaring all variables in advance, or any of these other organizational restrictions. They are sometimes carry-overs from times when it was considered normal for programmers to cater to the needs of their compilers. They are sometimes ideologically based decisions. They are not static types.  If one doesn’t want a language deciding how they should go about designing their code, it would be clearer to say so. Expressing this as a dislike for static typing confuses the issue.  This fallacy is often stated in different terms: “I like to do exploratory programming” is the popular phrase. The idea is that since everyone knows statically typed languages make you do everything up front, they aren’t as good for trying out some code and seeing what it’s like. Common tools for exploratory programming include the REPL (read-eval-print loop), which is basically an interpreter that accepts statements in the language a line at a time, evaluates them, and tells you the result. These tools are quite useful, and they exist for many languages, both statically and dynamically typed. They don’t exist (or at least are not widely used) for Java, C, or C++, which perpetuates the unfortunate myth that they only work in dynamically typed languages. There may be advantages for dynamic typing in exploratory programming (in fact, there certainly are some advantages, anyway), but it’s up to someone to explain what they are, rather than just to imply the lack of appropriate tools or language organization.  Fallacy: Dynamically typed languages provide no way to find bugs  A common argument leveled at dynamically typed languages is that failures will occur for the customer, rather than the developer. The problem with this argument is that it very rarely occurs in reality, so it’s not very convincing. Programs written in dynamically typed languages don’t have far higher defect rates than programs written in languages like C++ and Java.  One can debate the reasons for this, and there are good arguments to be had there. One reason is that the average skill level of programmers who know Ruby is higher than those who know Java, for example. One reason is that C++ and Java have relatively poor static type systems. Another reason, though, is testing. As mentioned in the aside above, the whole unit testing movement basically came out of dynamically typed languages. It has some definite disadvantages over the guarantees provided by static types, but it also has some advantages; static type systems can’t check nearly as many properties of code as testing can. Ignoring this fact when talking to someone who really knows Ruby will basically get you ignored in turn.  Fallacy: Static types imply longer code  This fallacy is closely associated with the one above about type declarations. Type declarations are the reason many people associated static types with a lot of code. However, there’s another side to this. Static types often allow one to write much more concise code!  This may seem like a surprising claim, but there’s a good reason. Types carry information, and that information can be used to resolve things later on and prevent programmers from needing to write duplicate code. This doesn’t show up often in simple examples, but a really excellent case is found in the Haskell standard library’s Data.Map module. This module implements a balanced binary search tree, and it contains a function whose type signature looks like this:  lookup :: (Monad m, Ord k) =&gt; k -&gt; Map k a -&gt; m a  This is a magical function. It says that I can look something up in a Map and get back the result. Simple enough, but here’s the trick: what do I do if the result isn’t there? Common answers might include returning a special “nothing” value, or aborting the current computation and going to an error handler, or even terminating the whole program. The function above does any of the above! Here’s how I compare the result against a special nothing value:  case (lookup bobBarker employees) of Nothing -&gt; hire bobBarker Just salary -&gt; pay bobBarker salary  How does Haskell know that I want to choose the option of getting back Nothing when the value doesn’t exist, rather than raising some other kind of error? It’s because I wrote code afterward to compare the result against Nothing ! If I had written code that didn’t immediately handle the problem but was called from somewhere that handled errors three levels up the stack, then lookup would have failed that way instead, and I’d be able to write seven or eight consecutive lookup statements and compute something with the results without having to check for Nothing all the time. This completely dodges the very serious “exception versus return value” debate in handling failures in many other languages. This debate has no answer. Return values are great if you want to check them now; exceptions are great if you want to handle them several levels up. This code simply goes along with whatever you write the code to do.  The details of this example are specific to Haskell, but similar examples can be constructed in many statically typed languages. There is no evidence that code in ML or Haskell is any longer than equivalent code in Python or Ruby. This is a good thing to remember before stating, as if it were obviously true, that statically typed languages require more code. It’s not obvious, and I doubt if it’s true.  Benefits of Static Types  My experience is that the biggest problems in the static\/dynamic typing debate occur in failing to understand the issues and potential of static types. The next two sections, then, are devoted to explaining this position in detail. This section works upward from the pragmatic perspective, while the next develops it into its full form.  There are a number of commonly cited advantages for static typing. I am going to list them in order from least to most significant. (This helps the general structure of working up to the important stuff.)  Performance  Performance is the gigantic red herring of all type system debates. The knowledge of the compiler in a statically typed language can be used in a number of ways, and improving performance is one of them. It’s one of the least important, though, and one of the least interesting.  For most computing environments, performance is the problem of two decades ago. Last decade’s problem was already different, and this decade’s problems are at least 20 years advanced beyond performance being the main driver of technology decisions. We have new problems, and performance is not the place to waste time.  (On the other hand, there are a few environments where performance still matters. Languages in use there are rarely dynamically typed, but I’m not interested enough in them to care much. If you do, maybe this is your corner of the type system debate.)  Documentation  If, indeed, performance is irrelevant, what does one look to next? One answer is documentation. Documentation is an important aspect of software, and static typing can help.  Why? Because documentation isn’t just about comments. It’s about everything that helps people understand software. Static type systems build ideas that help explain a system and what it does. They capture information about the inputs and outputs of various functions and modules. This is exactly the set of information needed in documentation. Clearly, if all of this information is written in comments, there is a pretty good chance it will eventually become out of date. If this information is written in identifier names, it will be nearly impossible to fit it all in. It turns out that type information is a very nice place to keep this information.  That’s the boring view. As everyone knows, though, it’s better to have self-documenting code than code that needs a lot of comments (even if it has them!). Conveniently enough, most languages with interesting static type systems have type inference, which is directly analogous to self-documenting code. Information about the correct way to use a piece of code is extracted from the code itself (i.e., it’s self-documenting), but then verified and presented in a convenient format. It’s documentation that doesn’t need to be maintained or even written, but is available on demand even without reading the source code.  Tools and Analysis  Things get way more interesting than documentation, though. Documentation is writing for human beings, who are actually pretty good at understanding code anyway. It’s great that the static type system can help, but it doesn’t do anything fundamentally new.  Fundamentally new things happen when type systems help computer programs to understand code. Perhaps I need to explain myself here. After all, a wise man (Martin Fowler, IIRC) one said:   “Any fool can write code that a computer can understand. Good programmers write code that humans can understand.”   I don’t disagree with Martin Fowler, but we have different definitions of understand in mind. Getting a computer to follow code step by step is easy. Getting a computer to analyze it and answer more complex questions about it is a different thing entirely, and it is very hard.  We often want our development tools to understand code. This is a big deal. I’ll turn back to Martin Fowler, who points this out as well  .  Correctness  Ultimately, though, the justification for static typing has to come back to writing correct code. Correctness, of course, is just the program doing “what you want.”  This is a really tough problem; perhaps the toughest of all. The theory of computation has a result called Rice’s Theorem, which essentially says this: Given an arbitrary program written in a general purpose programming language, it is impossible to write a computer program that determines anything about the program’s output. If I’m teaching an intro to programming class and assign my students to write “hello world”, I can’t program a grader to determine if they did so or not. There will be some programs for which the answer is easy; if the program never makes any I\/O calls, then the answer is no. If the program consists of a single print statement, it’s easy to check if the answer is yes. However, there will be some complicated programs for which my grader can never figure out the answer. (A minor but important technical detail: one can’t run the program and wait for it to finish, because the program might never finish!) This is true of any statement about programs, including some more interesting ones like “does this program ever finish?” or “does this program violate my security rules?”  Given that we can’t actually check the correctness of a program, there are two approaches that help us make approximations:   Testing: establishes upper bounds on correctness  Proof: establishes lower bounds on correctness   Of course, we care far more about lower bounds than upper bounds. The problem with proofs, though, is the same as the problem with documentation. Proving correctness is easy only somewhat insanely difficult when you have a static body of code to prove things about. When the code is being maintained by three programmers and changing seven times per day, maintaining the correctness proofs falls behind. Static typing here plays exactly the same role as it does with documentation. If (and this is a big if) you can get your proofs of correctness to follow a certain form that can be reproduced by machine, the computer itself can be the prover, and let you know if the change you just made breaks the proof of correctness. The “certain form” is called structural induction (over the syntax of the code), and the prover is called a type checker.  An important point here is that static typing does not preclude proving correctness in the traditional way, nor testing the program. It is a technique to handle those cases in which testing might be guaranteed to succeed so they don’t need testing; and similarly, to provide a basis from which the effort of manual proof can be saved for those truly challenging areas in which it is necessary.  Dynamic Typing Returns  Certainly dynamic typing has answers to this. Dynamically typed languages can sometimes perform rather well (see Dylan), sometimes have great tools (see Smalltalk), and I’m sure they occasionally have good documentation as well, though the hunt for an example is too much for me right now. These are not knock-down arguments for static typing, but they are worth being aware of.  The correctness case is particularly enlightening. Just as static types strengthened our proofs of correctness by making them easier and automatic, dynamic typing improves testing by making it easier and more effective. It simply makes the code fail more spectacularly. I find it amusing when novice programmers believe their main job is preventing programs from crashing. I imagine this spectacular failure argument wouldn’t be so appealing to such a programmer. More experienced programmers realize that correct code is great, code that crashes could use improvement, but incorrect code that doesn’t crash is a horrible nightmare.  It is through testing, then, that dynamically typed languages establish correctness. Recall that testing establishes only upper bounds on correctness. (Dijkstra said it best: “Program testing can be used to show the presence of bugs, but never to show their absence.“) The hope is that if one tries hard enough and still fails to show the presence of bugs, then their absence becomes more likely. If one can’t seem to prove any better upper bound, then perhaps the correctness really is 100%. Indeed, there is probably at some correlation in that direction.  What is a Type?  This is as good a point as any to step back and ask the fundamental question: what is a type? I’ve already mentioned that I think there are two answers. One answer is for static types, and the other is for dynamic types. I am considering the question for static types.  It is dangerous to answer this question too quickly. It is dangerous because we risk excluding some things as types, and missing their “type” nature because we never look for it. Indeed, the definition of a type that I will eventually give is extremely broad.  Problems with Common Definitions  One common saying, quoted often in an attempt to reconcile static and dynamic typing, goes something like this: Statically typed languages assign types to variables, while dynamically typed languages assign types to values. Of course, this doesn’t actually define types, but it is already clearly and obviously wrong. One could fix it, to some extent, by saying “statically typed languages assign types to expressions, ...” Even so, the implication that these types are fundamentally the same thing as the dynamic version is quite misleading.  What is a type, then? When a typical programmer is asked that question, they may have several answers. Perhaps a type is just a set of possible values. Perhaps it is a set of operations (a very structural-type-ish view, to be sure). There could be arguments in favor of each of these. One might make a list: integers, real numbers, dates, times, and strings, and so on. Ultimately, though, the problem is that these are all symptoms rather than definitions. Why is a type a set of values? It’s because one of the things we want to prove about our program is that it doesn’t calculate the square roots of a string. Why is a type a set of operations? It’s because one of the things we want to know is whether our program attempts to perform an impossible operation.  Let’s take a look at another thing we often want to know: does our web application stick data from the client into SQL queries without escaping special characters first? If this is what we want to know, then these become types. This article by Tom Moertel  builds this on top of Haskell’s type system. So far, it looks like a valid definition of “type” is as follows: something we want to know.  A Type System  Clearly that’s not a satisfactory definition of a type. There are plenty of things we want to know that types can’t tell us. We want to know whether our program is correct, but I already said that types provide conservative lower bounds on correctness, and don’t eliminate the need for testing or manual proof. What makes a type a type, then? The other missing component is that a type is part of a type system.  Benjamin Pierce’s book Types and Programming Languages  is far and away the best place to read up on the nitty gritty details of static type systems, at least if you are academically inclined. I’ll quote his definition.   A type system is a tractable syntactic method for proving the absence of certain program behaviors by classifying phrases according to the kinds of values they compute.   This is a complex definition, but the key ideas are as follows:   syntactic method .. by classifying phrases : A type system is necessarily tied to the syntax of the language. It is a set of rules for working bottom up from small to large phrases of the language until you reach the result.  proving the absence of certain program behaviors : This is the goal. There is no list of “certain” behaviors, though. The word just means that for any specific type system, there will be a list of things that it proves. What it proves is left wide open. (Later on in the text: “... each type system comes with a definition of the behaviors it aims to prevent.“)  tractable : This just means that the type system finishes in a reasonable period of time. Without wanting to put words in anyone’s mouth, I think it’s safe to say most people would agree that it’s a mistake to include this in the definition of a type system. Some languages even have undecidable type systems. Nevertheless, it is certainly a common goal; one doesn’t expect the compiler to take two years to type-check a program, even if the program will run for two years.   The remainder of the definition is mainly unimportant. The “kinds of values they compute” is basically meaningless unless we know what kinds we might choose from, and the answer is any kind at all.  An example looks something like this. Given the expression 5 + 3 , a type checker may look at 5 and infer that it’s an integer. It may look at 3 and infer it’s an integer. It may then look at the + operator, and know that when + is applied to two integers, the result is an integer. Thus it’s proven the absence of program behaviors (such as adding an integer to a string) by working up from the basic elements of program syntax.  Examples of Unusual Type Systems  That was a pretty boring example, and one that plays right into a trap: thinking of “type” as meaning the same thing it does in a dynamic type system. Here are some more interesting problems being solved with static types.   http:\/\/wiki.di.uminho.pt\/twiki\/pub\/Personal\/Xana\/WebHome\/report.pdf  . Uses types to ensure that the correct kinds of data are gotten out of a relational database. Via the type system, the compiler ends up understanding how to work with concepts like functional dependencies and normal forms, and can statically prove levels of normalization.  http:\/\/www.cs.bu.edu\/~hwxi\/academic\/papers\/pldi98.pdf  . Uses an extension to ML’s type system to prove that arrays are never accessed out of bounds. This is an unusually hard problem to solve without making the languages that solve it unusable, but it’s a popular one to work on.  http:\/\/www.cis.upenn.edu\/~stevez\/papers\/LZ06a.pdf  . This is great. This example uses Haskell’s type system to let someone define a security policy for a Haskell program, in Haskell, and then proves that the program properly implements that security. If a programmer gets security wrong, the compiler will complain rather than opening up a potential security bug in the system.  http:\/\/www.brics.dk\/RS\/01\/16\/BRICS-RS-01-16.pdf  . Just in case you thought type systems only solved easy problems, this bit of Haskell gets the type system to prove two central theorems about the simply typed lambda calculus, a branch of computation theory!   The point of these examples is to point out that type systems can solve all sorts of programming problems. For each of these type systems, concepts of types are created that represent the ideas needed to accomplish this particular task with the type system. Some problems solved by static type systems look nothing like the intuitive idea of a type. A buggy security check isn’t normally considered a type error, but only because not many people use languages with type systems that solve that problem.  To reiterate the point above, it’s important to understand how limiting it is to insist, as many people do, that the dynamic typing definition of a “type” is applied to static typing as well. One would miss the chance to solve several real-world problems mentioned above.  The True Meaning of Type  So what is a type? The only true definition is this: a type is a label used by a type system to prove some property of the program’s behavior. If the type checker can assign types to the whole program, then it succeeds in its proof; otherwise it fails and points out why it failed. This is a definition, then, but it doesn’t tell us anything of fundamental importance. Some further exploration leads us to insight about the fundamental trade-offs involved in using a static type checker.  If you were looking at things the right way, your ears may have perked up a few sections back, when I said that Rice’s Theorem says we can’t determine anything about the output of a program. Static type systems prove properties of code, but it almost appears that Rice’s Theorem means we can’t prove anything of interest with a computer. If true, that would be an ironclad argument against static type systems. Of course, it’s not true. However, it is very nearly true. What Rice’s Theorem says is that we can’t determine anything. (Often the word “decide” is used; they mean the same thing here.) It didn’t say we can’t prove anything. It’s an important distinction!  What this distinction means is that a static type system is a conservative estimate. If it accepts a program, then we know the program has the properties proven by that type checker. If it fails... then we don’t know anything. Possibly the program doesn’t have that property, or possibly the type checker just doesn’t know how to prove it. Furthermore, there is an ironclad mathematical proof that a type checker of any interest at all is always conservative. Building a type checker that doesn’t reject any correct programs isn’t just difficult; it’s impossible.  That, then, is the trade-off. We get assurance that the program is correct (in the properties checked by this type checker), but in turn we must reject some interesting programs. To continue the pattern, this is the diametric opposite of testing. With testing, we are assured that we’ll never fail a correct program. The trade-off is that for any program with an infinite number of possible inputs (in other words, any interesting program), a test suite may still accept programs that are not correct - even in just those properties that are tested.  Framing the Interesting Debate  That last paragraph summarizes the interesting part of the debate between static and dynamic typing. The battleground on which this is fought out is framed by eight questions, four for each side:   For what interesting properties of programs can we build static type systems?  How close can we bring those type systems to the unattainable ideal of never rejecting a correct program?  How easy can it be made to program in a language with such a static type system?  What is the cost associated with accidentally rejecting a correct computer program?  For what interesting properties of programs can we build test suites via dynamic typing?  How close can we bring those test suites to the unattainable ideal of never accepting a broken program?  How easy can it be made to write and execute test suites for programs?  What is the cost associated with accidentally accepting an incorrect computer program?   If you knew the answer to those eight questions, you could tell us all, once and for all, where and how we ought to use static and dynamic typing for our programming tasks.  ","title":"What to Know Before Debating Type Systems","url":"\/articles\/what-to-know-before-debating-type-systems.html"},{"body":"    History  Variables and Types  Classes are Types  The Problem with :builder  Builder Guidelines     I’ve been spending a lot of time designing a new OO syntax for the Perl language  called Corinna. This syntax is not designed to be another object-oriented module. Instead, it’s designed to be a be a core part of the language. It’s received a lot of support (and a few detractors), but I have a problem with it.  Lately I’ve been worried about the slot variable :builder attribute and I’ve been concerned that it’s problematic. I’ve nattered on about this on IRC and just generally let the concerns simmer in my mind. If we’re going to get to propose something for the core, it’s better that we avoid mistakes before that time rather than after. Then Damian Conway shared some thoughts with me about his concerns for :builder and that was the final straw. Something has to change.   History  As part of our proposal, we have this:  class SomeClass { has $x :builder; method _build_x () { return rand; } }  If you call SomeClass-&gt;new , you’ll get a random numeric value assigned to the $x slot variable. Note that $x has no reader or writer, so it’s designed to be completely encapsulated. But it’s not and that’s a problem.  That’s been bugging me for quite a while, but to explain that, I need to go back in time.  In 1967 (my birth year, but I swear it’s a coincidence), the first truly class-based OO language was released, Simula 67  and it introduced classes, polymorphism, encapsulation, and inheritance (and coroutines (!), but let’s not go there).  Classes, polymorphism, and encapsulation are relatively uncontroversial, but the debate over inheritance has raged for decades.  Alan Kay, the man who invented the term “Object Oriented” in Utah “sometime after 1966”  and who’s been doing OO programming before most of us were born, has made it clear over the years that inheritance in OO is very problematic. He’s even left it out of the implementation of some OO languages he’s created. And then I found a Quora question entitled “What does Alan Kay think about inheritance in object-oriented programming?” and someone purporting to be Alan Kay responded  with a deeply thoughtful answer. His first paragraph is fantastic (emphasis mine):   Simula I didn’t have inheritance (paper ca 1966) and Simula 67 did (paper ca 1968 or so). I initially liked the idea — it could be useful — but soon realized that something that would be “mathematically binding” was really needed because the mechanism itself let too many semantically different things to be “done” (aka “kluged”) by the programmer. For example, there is no restriction of any kind to have a subclass resemble a superclass, be a refinement of a superclass, etc. All relies on the cleanliness of mind of programmers (and even the most clean of these often just do things they need when in the throes of debugging).   In other words, if I create a class named Vehicle::Motorized , there is nothing to stop your Order::Customer class from inheriting from it. Would that make sense? Probably not. But there we are.  Next, let’s take a look at a Python object:  class Point: def __init__(self, x, y): self.x = x self.y = y def inverse(self): return Point(self.y,self.x) point = Point(7,3.2).inverse() print(point.x) point.x = &quot;foo&quot;  And for those who would like to see that in our current proposal:  class Point { has ($x,$y) :reader :writer :new; method inverse { return $class-&gt;new( x =&gt; $y, y =&gt; $x ); } } my $point = Point-&gt;new( x =&gt; y =&gt; 3.2 )-&gt;inverse; say $point-&gt;x; $point-&gt;x(&quot;foo&quot;);  As you can see, we’ve assigned “foo” to the x value of a Point object. We’ve done this in both Python and our new OO syntax for Perl. Sadly, due to the need to harmonize types across the language, we can’t have type constraints in v0.1.0. This is a serious limitation, but we’re taking baby steps.  But what about the Python version? Surely they must validate their data, right? No. Pythonistas have assured me that validating your data is very unpythonic  .  (I envision spit-takes from tons of Moo\/se fans out there right now).  Python 3.7 contains data classes  and those will help, but I don’t know how widespread the usage is.   Variables and Types  But what does that have to do with inheritance?  For many languages, you can try something like (example in the Go programming language) var myvar float64 = \"foo\" and it will throw an exception. It might be compile-time. It might be runtime. But it will die a horrible death because the variable, myvar , has a type associated with it and if you assign something that doesn’t match that type (or that the language can’t\/won’t coerce), then things go boom.  Why is that? Because on line 372 of your whopping huge function , it’s easy to have something like myvar := stuff[0:2] and not be sure what kind of data stuff[0:2] contains.  So we can say that—for a small subset of types the programming language can understand—variables in those kinds of languages are the “experts” about what they can do and if you try to put that variable in an invalid state, it blows up.  That’s great. But what about var celsius float64 = -1300.15 . Is that valid?  Absolute zero is -273.15 on the Celsius scale, so we can make a reasonable argument that the above is not valid. In languages like Raku  , we easily fix that:  subset Celsius of Num where * &gt;= -273.15; my Celsius $temp = -1300.15;  The above will fail at compile-time. Most languages that allow types, however, aren’t quite as flexible, but you can use classes instead.  Classes are Types  my $temp = Celsius-&gt;new( temperature =&gt; -1300.15 );  Assuming your class is sane, the above should throw an exception because you’re trying to create an object with an invalid state.  If you think that’s similar to how built-in types work, it is. We can think of a class as a user-defined type. Types have a set of allowed values and operators you can apply to those values. Classes have (often complex) sets of allowed values and methods (operators) you can apply to those values.  Just as an int is often an expert in values -32,768 to 32,767, or -2,147,483,648 to 2,147,483,647 if you’re using 4 bytes (larger values for more bytes, of course), so is your Celsius class an expert about allowed Celsius temperatures. And your Personal Shopper knows exactly how much money it can spend, and even if it knows you have enough money to buy cocaine, it’s still going to throw an exception if you request that because it’s an illegal operation.    The Problem with :builder  And this gets us to :builder . In Perl‘s Moo\/se family of OO, you can do something similar to this (but using our new syntax):  class Parent { has $x :builder; method _build_x () { return rand; } } class Child isa Parent { method _build_x () { return &quot;not a number&quot; } }  This has exactly the same issue with see with Python’s point.x = \"foo\" . It’s hardly necessary to belabor the point, but I will: $x is supposed to be private, but we’ve grossly violated encapsulation.  Ah, you reply. But Moo\/se has types. So let’s look at Moo\/se, using a better example (not great, but enough to get the point across):  package Collection { use Moose; # this is a personal module which gives me a &quot;saner&quot; Perl # environment use Less::Boilerplate; use Types::Standard qw(Int ArrayRef); has _index =&gt; ( is =&gt; 'rw', isa =&gt; Int-&gt;where('$_ &gt;= 0'), builder =&gt; '_build__index', ); # default, but maybe someone wants a different default sub _build__index { 0 } has items =&gt; ( is =&gt; 'ro', isa =&gt; ArrayRef, required =&gt; 1, ); sub BUILD ( $self, @ ) { my @items = $self-&gt;items-&gt;@*; my $type = ref $items[0]; foreach my $item (@items) { if ( not defined $item ) { croak(&quot;items() does not allow undefined values&quot;); } if ( ref $item ne $type ) { croak(&quot;All items in collection must be of the same type&quot;); } } } sub num_items ($self) { return scalar $self-&gt;items-&gt;@*; } sub next ($self) { my $i = $self-&gt;_index; return if $i &gt;= $self-&gt;num_items; $self-&gt;_index( $i + 1 ); return $self-&gt;items-&gt;[$i]; } sub reset ($self) { $self-&gt;_index(0); } }  It goes without saying that writing an iterator like the above can be rather problematic for a number of reasons, and yes, the _build__index method is kind of silly, but it shows the issue. For example, what would a negative index do? In the above, it would throw an exception because of the Int-&gt;where( '$_ &gt;= 0' ) constraint. But what would happen if we set the index to a value larger than the size of the items array reference (and ignoring that the above allows $coll-&gt;_index($integer) ).  package My::Collection { use Moose; extends 'Collection'; sub _build__index { 5 } } my $collection = My::Collection-&gt;new( items =&gt; [qw\/foo bar baz\/] ); while ( defined( my $item = $collection-&gt;next ) ) { say $item; }  What do you think the above does? The constraint passes, but it prints nothing. That’s because the allowed value of the index is tightly coupled to something else in that class, the number of items in the collection.  Oh, but Ovid, I wouldn’t write something like that!  Yes, you would. I do it too. We all do it when we’re quickly hacking on that 883 line class from our 1,000 KLOC codebase and we’re rushing to beat our sprint deadline. Programmers do this all the time because we have a language which allows this, but OO modules which encourage this. You can inspect tons of OO modules and find attributes (a.k.a “slot variables” in our new system) which are coupled with other values. We like to minimize that, but sometimes we can’t and classes are supposed to encapsulate this problem and handle it internally.  My suggestion is that, if you want to allow this internally, you have to do it manually with the ADJUST phaser  .  class Collection { has $index; has $items :new; ADJUST (%args) { # other checks here $index = $self-&gt;_default_index; if ( $index &gt; $items-&gt;@* - 1 ) { # don't let subclasses break our code croak(...); } } method _default_index () { 0 } # other methods here }  Do I recommend that? No. Can you do it? Yes. And it has some advantages.  First, you are being explicit about which slots are can access in your subclasses. Second, you have fine-grained control over slot initialization ordering.  By default, with Corinna, slots are initialized in the order declared (until you hit :new , in which case all :new slots are initialized at once). In Moo\/se it’s harder to predict the order of initialization. That’s why you often see a bunch of attributes declared as lazy to ensure that non-lazy attributes are set first. If you have circular lazy dependencies, it can be hard to work out and you fall back to BUILD or BUILDARGS as needed.  Yes, this does mean a touch more work on the off chance you need to override a slot in a subclass. I’ve been writing Moo\/se for years and I can count on the fingers of one hand the number of times I’ve needed to do this. However, I can’t count how many times I’ve unnecesssarily littered my namespace with sub _build... methods because this is largely what Moo\/se prefers and what developers will call you out on in code reviews if you do it some other way.  Oh, and we also hit issues like this. Which of these is correct?  class Position { slot $x :name(ordinate) :builder; # and many lines later... method _build_ordinate { return rand } # Does the builder follow the public slot name? } class Position { slot $x :name(ordinate) :builder; # and many lines later... method _build_x { return rand } # Or does it follow the private slot variable name? }  If we were to allow builders, we should, at least, mandate the syntax :builder($method_name) and not allow default builder names.  Builder Guidelines  So here are a few guidelines we should follow for assigning default values to slots.   Every slot should have a default value.   It is often a code smell to have an undefined slot.   Use the = default syntax   In the example below, :new says we can pass the value to the constructor, but if we don’t, it will get the value 42.  has $x :new = 42;   YAGNI (You Ain’t Gonna Need It).   Don’t allow overriding a private slot value unless you have a clear need for it. This also prevents littering the namespace with a _build_... methods.   Liskov is your friend   Remember the Liskov Substitution Principle  : if you have a subclass and it must be allowed to override some of this data, remember that a subclass should be a more specialized version of a parent class and must be usable anywhere the parent can be.   Check for coupling   If the slot data is tightly coupled to some other slot data, consider breaking the coupling or ask yourself if delegation is more appropriate. Have checks in BUILD (Moo\/se) or ADJUST (Corinna) to verify the integrity of your class.  If all of the above seems like too much to remember, just don’t allow child classes to set parent slot data. Corinna should follow one of the early principles guiding its design: you should be allowed to do bad things, but the language design shouldn’t encourage bad things.  ","title":"Classes Should Not Override Parent Attributes","url":"\/articles\/the-problem-with-builder.html"},{"body":" It's safe to say that out of all of the software development project problems I've come across, there's one I don't know how to solve: the tyranny of budgets. If you can provide any insight, please leave a comment below. I'd love to hear it.  A Typical Horror Story  This following is a story I've told a few times, so bear with me if you've heard it before.  A large European company was earning millions of euros (a low seven figures) from their core software and was considered a \"market leader\" in what they do. Their software was written in Perl, long ago, by developers who didn't understand Perl or software development. Those software developers were long gone. Thus, after well over a decade of \"how can we squeeze this in\" development, they had a steaming pile of ones and zeros that no self-respecting software developer would want to work on.  Fortunately, I am no self-respecting software developer. I like rebuilding legacy systems the same way my friend James loves rebuilding old cars. Except that the company didn't call me to rebuild their software. They decided that this highly profitable software was going to be thrown away and rewritten in Java. They could find more developers and the code would be \"cleaner.\" Except, as many companies discover far too late, rewrites are often financial disasters that drag on forever . The old code base had a ton of features that were time-consuming to port over and by the time they called me to \"maintain\" the Perl code, a couple of years had gone by and the company only had two customers, out of hundreds, using the new system.  Given that I have a lot of experience working with legacy codebases, the initial conversations went well. I clearly understood their problem domain, I am a deep expert in this area, and I was able to identify several of their issues before I had even seen the code   They were sold; I was the person who was going to help stabilize the code base earning them millions every year.  And then the trainwreck happened.  They wanted to have me part-time, on call, and pay me a junior developer's rates.  Excuse me? You're unable to maintain your core product and you're looking for a part-time junior dev for this?  \"We're phasing this product out and that's all we have in our budget.\"    A year later, they called me back. It seems the $100\/day developer they found wasn't that good of a deal after all and he eventually stopped returning their phone calls. They still didn't have the new system ready and estimated it would take at least a year before it could properly launch. Would I be interested in working with them on the legacy system?  Of course I would. This happens all the time in consulting and you either accept it or go back to being an employee.  Then they explained they wanted to have me part-time, on call, and pay me a junior developer's rates.  Fast forward to today, several years later. I check their web site and see they're still in business, but curiously, no mention is made of the core business they were in. They seem to have shifted into a related field, cunningly using the word \"AI\" in its description.   . Maybe it was a brilliant business decision? Maybe it was a Hail Mary pass  ? Maybe it was a gradual evolution? Who knows? But I suspect they didn't throw away their multi-million euro core product on a whim.  Used Cars  I've always been amazed at how many companies treat their software like a college student buying a used car. They have a limited budget. They buy that old Ford Mustang for $1,500. They complain bitterly at their maintenance costs. And their fuel costs. And their insurance costs!   And the time missed from school\/work dealing with said costs.  In fact, I was the one who bought that old Mustang and it was a beast. Its fuel consumption should have been measured in gallons per mile, it pissed oil all over the place and, thanks to an exhaust leak from a warped manifold, it kept burning through the wires to the starter, causing me to replace the starter every few months.   That $1,500 was all I thought I could afford but it cost me a hell of a lot more than a more expensive car.  But you know what? I didn't have much of a choice. Local public transportation was a crap shoot, meaning I risked missing work and losing my job. The distance between home, work, and college meant I couldn't walk or ride a bike and I could not afford to move. It was a cheap car or drop out of school. It was all I could afford and I couldn't afford it.  Software Projects  But there is a critical difference between used cars and software projects.   . It's easy to get another car, if you have the money. The same isn't true for custom software. Maybe you get a great deal on that used car or maybe it breaks down every few months and has to be repaired. Maybe the engine block cracks and you have to junk the car. Do you really want to roll those dice on your software?  The answer, for many companies, is \"yes.\" More than once I've heard decision makers say \"this makes perfect sense and would save us a lot of money in the long run, but in the short run, we don't have the budget for it.\"  And you know what? I get that. While it's often penny wise and pound foolish, sometimes companies just don't have the money. Or the project isn't that important to the company.   . Or, as in the case of the company that kept trying to hire me part-time, sometimes it's simply inertia: \"we've made a decision and now we don't have to think about it.\"  And the last reason is why so much of our software is rubbish.  I honestly don't know how to break a company out of that thinking, so honestly, we don't even try. We politely explain our point of view, and move on to clients who understand that buying a cheap used car is a crap shoot. But I wish I could figure out how to get more people to read Douglas Hubbard and learn that development costs are the least important aspect of estimating project value .  Today, there's a Volkswagen Golf in front of our house. It cost more to buy than that Mustang did, but it costs far less per month and will do so for a long, long time. It was a good choice.  ","title":"The Tyranny of Budgets","url":"\/articles\/the-tyranny-of-budgets.html"},{"body":" Note: the following details a serious, expensive class of bugs that your project may have, but that many don't know about. Out of necessity, I'll lightly touch on technical details.  I was hiring developers for a large, complicated application. I happened to mention to one of the more promising candidates that the application was fairly write-heavy and we might experience some performance concerns later. This developer, with years of experience and skills across multiple technologies that might have been of use to us, replied \"you're going to either need to shard your database or switch to NoSQL.\"  That was enough to guarantee they didn't get the job.  It's a distressingly common problem: with absolutely no understanding of how the database was designed or what the application did, an experienced developer made a strong declaration about how to fix a problem we didn't yet have. In fact, when companies hire me for consulting, one of the first things I do is examine the data storage layer. It's often in pretty bad shape and a constant source of bugs because developers learn SQL and think they understand databases.  If you take nothing else from this article, take this: learning how to write and optimize SQL has as much in common with designing a database as being an expert mechanic qualifies you to design a car .  Data Quality Woes  In many startups, there's not a lot of up front money, so a qualified DBA is viewed as a luxury rather than a necessity. As a result, the devs choose a data storage tool based on what they're comfortable with instead of what the project actually requires. MySQL is so popular and easy to use that many developers choose it out of habit, tune a few parameters for performance, and start developing.  As the project grows, data issues are noticed and get fixed on an ad hoc basis. Variants of the following bug abound:  mysql&gt; create table example (tooshort varchar(3)); Query OK, 0 rows affected (0.00 sec) mysql> insert into example (tooshort) values (\"12345\"); Query OK, 1 row affected, 1 warning (0.00 sec) mysql&gt; select tooshort from example; +----------+ | tooshort | +----------+ | 123 | +----------+  As you can see from the above, MySQL happily truncated the data, throwing away valuable information. It issued a warning, but most applications ignore database warnings. For one of our clients, we found two years of analytic data had to be thrown away because of this bug, even though this warning was issued on every insert.  Once developers learn about MySQL's strict mode, they often try to enable it, only to watch the application crash and burn because it was written with the assumption that division by zero produces NULL values instead of errors, \"0000-00-00\" is a real date, and many other tiny problems. These same applications usually lack foreign key constraints, or use them inconsistently. And more than one developer has solved deadlock issues by not using database transactions.  By the time strict mode is finally enabled, there's a layer of bugs in the database and a layer of related \"strict mode workarounds\" in the application level. It becomes expensive to maintain and much development effort is spent on correcting existing problems rather than build new features that could earn new business.  Those who use Oracle or SQL Server often have databases just as bad, so this isn't just a \"MySQL thing.\" Curiously, we've noticed that our clients who use PostgreSQL tend to have better quality databases, possibly stemming from a database community which is obsessed with data quality.  Performance Woes  For one company, a domain name registrar, I was helping them work on fraud prevention tools when a project manager mentioned that a particular query took 15 minutes to return. The team lead explained that there was \"too much data\" to query efficiently. That \"too much data\" was only three million records. After running explain on the query, I discovered they weren't using indexes. A few minutes later, that 15 minute query took under two seconds to run. Were it not for the accident of my being there, they would have concluded the database was \"too slow\" and reached for a NoSQL solution.  Another common issue, endemic with ORMs, is fetching multiple columns, only to use two or three of them. Or discovering that MySQL often forces a disk sort when fetching rows with text or blob fields, even if you're not sorting on those fields. Or constantly updating only one column in a table when 20 other columns that never change, causing a lot of unexpected disk I\/O. For these and many other issues, most of which are easy to solve, devs who are unfamiliar with databases often reach for NoSQL solutions without understanding why they had a problem in the first place. Quite often they've traded a set of known problems for a set of unknown problems. This is not a good business decision.  More to the point: before relational databases, all databases were NoSQL. That's why we have relational databases.  Data is King  So why is this a big deal? Well, the term \"information economy\" isn't bandied about for nothing. Consider that most applications have four phases, though their structure might obscure this.   Initialization  Input  Calculation  Output   Virtually every application has those phases, but what do they mean? Simply put:   Initialization of data  Input of data  Calculation of data  Output of data   Your application is all about data. For many people in management, they see the app or the web page, but they don't see the database hidden underneath. They understand that without their data the company is probably in serious trouble, but they don't give much thought to it. In fact, while total data loss will probably bankrupt your company, few give much thought to the the standard data quality issues that are a constant financial drain because it's assumed to be part of the cost of doing business. And the developers who don't understand how to use databases often tell management \"that's just the way this technology works.\"  If there is one thing to take away from this article, it's to know that a properly designed database will still allow you to add incorrect data such as a misspelled first name, but it makes it hard to add invalid data, such as an order which references a non-existant customer, or a negative price for a product. Data maintenance costs drop significantly when you have a great database.  Even the barest minimum of work understanding how to design databases can tremendously improve quality. One of my most popular talks—rated 5 out of 5 stars at OSCON—is \"How to Fake a Database Design.\"     You don't need to be an expert in databases to do a much better job of using them. First, learn the basics of database design. Understand that foreign key constraints, check constraints, and other tools are part of that design, and not just \"nice to have\" features. Once you have your data quality ensured, then you can worry about performance. As the old adage goes \"first make it work, then make it fast.\"  NoSQL, sharding, and other techniques to get performance wins can wind up costing you a lot of money in the long run. Until you really understand why your current database isn't fast enough, don't reach for the shiny tools. You'll thank me later.  As an aside: I use and recommend NoSQL solutions from time to time (a shout-out to Redis: I love Redis). I am not \"anti-NoSQL.\" I simply find that by properly designing databases and understanding the underlying causes of database issues, NoSQL solutions often offer us little value.  ","title":"How the Database Can Hurt Your Startup","url":"\/articles\/how-databases-can-hurt-your-startup.html"},{"body":" Update: Many peoplea argue that messaging is flawed because your software breaks if you don't get a response. I wrote a new article, begging to differ .  I'm not going to link to the thread which set me off on this topic because there's no need to embarrass the people who say things like ...  Programming existed. Objects existed. Then [Alan] Kay contributed \"-oriented\" to the term \"object-oriented programming\" and the world went wild, but why?  Or another person, objecting to Kay's criticism of \"modern\" OO programming:  \"It's different than what I already know so it's wrong.\" This kind of thinking is why I quit CS major, chose to not pursue a career in programming, and haven't looked back.  This was largely in response to Alan Kay's famous 1997 OOPSLA keynote which is worth watching in its entirety.     There's much more in that thread, and while some understood what Kay was trying to say, many others seemed unaware of Kay's background and acted as if he was some grumpy old dude who just graduated from a Rails Bootcamp.  This is Dr. Alan Kay we're talking about! He doesn't have random opinions about \"objects\", he invented the word back in the 60s  . He saw what was happening in the programming world and was helping to craft many of these ideas, so he created a word to make these ideas easier to talk about.  And he's still bitter about choosing that word. It made people focus on the implementation rather than the behavior and it's all been downhill from there. Or as Dr. Kay put it \"I invented the term Object-Oriented, and I can tell you I did not have C++ in mind.\"  Today, most developers think OOP is about classes and inheritance. Some of the bright ones pipe up about encapsulation and polymorphism, but that's about it. class Dog isa Mammal {...} has won.  But for Dr. Kay, he states that OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things.  Why? Well, part of his background was cell biology and when he did the math on their computational power, he realized that while software routinely has trouble scaling, cells can easily coordinate and scale by a factor of over a trillion, creating some of the most fantastically complex things in existence, capable of correcting their own errors. By comparison, the most sophisticated computer software programs are slow, tiny, bugfests. Kay's conception of OOP starts with a single question: how can we get our software to match this scalability?  Isolation  First, let's discuss isolation. This is a shorter term than \"local retention and protection and hiding of state-process\".  The interior of a cell is messy and confusing, but the cell membrance wraps this up in a tidy package, hiding the internal details. It's estimated that around 50 to 70 billion cells die in your body every day. But you don't. Could your software keep running if you had millions of exceptions being thrown every minute? I doubt it.  You not dying when your cells die isn't encapsulation; it's isolation. Consider the following (awful) example:  class MyExample: def reciprocal(self, num): return 1.0\/num example = MyExample() print example.reciprocal(4); print example.reciprocal(0);  In the above code, we've encapsulated the reciprocal equation in the class, but then ...  0.25 Traceback (most recent call last): File \"class.py\", line 7, in &lt;module&gt; print example.reciprocal(0); File \"class.py\", line 3, in reciprocal return 1.0\/num ZeroDivisionError: float division by zero  The object dies, as does the code which contained it. This is the antithesis of what Dr. Kay is trying to get us to understand.  If you think of Web browsers and servers as objects, however, we see something closer to his vision.   If your browser crashed every time a web server crashed or was otherwise unavailable, Microsoft IIS would never have reached 2.0.  Now that we sort of understand a core idea of Kay's, Lets take it further. Kay points out that from the early days of Arpanet in the 60s, to the time of his OOPSLA keynote in 1997, Arpanet had grown roughly 100 million times the size of what it was.   And it didn't have to be repeatedly taken down for maintenance every time we wanted to extend it. The internet, today, is sometimes cited by Kay as the only working example of his OO model.    Extreme Late Binding  Another key point is his arguing for extreme late-binding. What does that mean? Well, consider this code:  my $order = OrderFactory->fetch(%order_args); my $invoice = $order->invoice;  If you have multiple \"order\" classes, you may not know which class you are dealing with, so you can't know, at compile time which invoice method you're calling. OOP languages generally don't select (bind) the method ( invoice ) for the invocant ( $order ) until run time. Otherwise, polymorphism can't work.  But what's extreme late binding? Does the invoice method exist? In a language like Java, that code won't even compile if the method doesn't exist. It might even be dead code that is never called, but you can't compile it if that method isn't there. That's because Java at least checks to ensure that the method exists and can be called.  For many dynamic languages, such as Perl (I &#9829; Perl), there's no compilation problem at all because we don't bind the method to the invocant until that code is executed, but you might get a panicked 2AM call that your batch process has failed ... because you might have encapsulation, but not isolation. Oops.   . This is \"extreme\" late binding with virtually no checks (other than syntax) performed until runtime.  Binding can also refer to binding a variable type to data.   . While this is a grotesque oversimplification of a complex argument, one way of looking at the difference between static and dynamic languages is that static languages such as Java often bind the data type to the variable (in other words, the named container of the data), meaning that you can never assign a different type to that variable, while dynamic languages such as Perl bind the data type to the data itself, meaning that the data knows about its type, but the variable often does not.   The latter is late binding because you often cannot infer, from static analysis, the data types at compile time.    Extreme late-binding is important because Kay argues that it permits you to not commit too early to the \"one true way\" of solving an issue (and thus makes it easier to change those decisions), but can also allow you to build systems that you can change while they are still running ! When was the last time you changed the behavior of a program and didn't have to stop it and restart it, often waiting hours for a complex recompilation? If you've worked with Smalltalk or Erlang you've probably had this pleasure, but most of the rest of us are still struggling with the hell of Makefiles and scheduled downtimes.  Messaging  We can kind of see the isolation and late-binding, but it's messaging where Kay's message seems to fall down for most, even though he's quite insistent that it's the most important thing there is in OOP. In essence, objects should be able to announce that they did things and other objects can ignore them or say \"hey, that's cool. Here's what I did in response.\"  But that doesn't really get to the core concept of messaging and frankly, this is the one area where I think Kay has been a touch vague, largely because many developers think that ideas are nice, but they want to see an implementation or proof of concept. In Kay's world. every object is almost like an entire computer, not just a unique area of responsibility, and each object can receive messages and figure out whether or not it knows how to deal with them.  In other words, you don't execute code by calling it by name: you send some data (a message) to an object and it figures out which code, if any, to execute in response. In fact, this can improve your isolation because the receiver is free to ignore any messages it doesn't understand. It's a paradigm most are not familiar with, but it's powerful.  There's a lot more to be said about this, but I'll stop and instead direct you to this article, written by Alan Kay, about The Early History of Smalltalk  . I actually found myself giddy reading that, getting a glimpse into the Cambrian explosion  of computing ideas which happened during the 60s and 70s. Sadly, computing, as with evolution, left that explosive era with powerful tools, but fewer ideas. Functional programming is slowly gaining more converts, logic programming is largely ignored   , but most programming today is procedural or class-based OOP. But Kay's ideas of what he intended for OOP are still with us. You can check out Erlang  and discover a marvelous programming language where code is isolated, late-binding, and passes messages around. And what do you get for that?   Extremely fault-tolerant code  High availability  Code whose behavior can be changed while the software is still running   That last point is a killer feature for many companies. The Erlang syntax and behavior is strange for many developers (it looks a lot like Prolog with the solver removed), if you have millions of customers counting on your service to always be there, Erlang is a powerful option.  OOP Today  Kay argues (correctly, IMHO), that the computer revolution hasn't happened yet because while our bodies are massively scalable meat computers, our silicon computers generally don't scale in the slightest. This isn't just because silicon is slow; it's because of things like print customers.fetch(customer_id).last_name not actually having a last_name method, throwing an exception (assuming it compiled in the first place) and programmers getting frantic late-night calls to bring the batch system back up. The only real upside is that it offers job security.  Sadly, this is where most of us are today. And would-be programmers get silly lectures telling them that class Motorcycle extends Vehicle , the instructor ( isa Person ) gets flustered explaining for the n th time the difference between a class and an instance, while a student ( isa Person ) in the back is noticing that a bicycle looks an awful lot like a motorcycle and the code she's looking at doesn't seem to account for this.  So class-based OO has won,   even with its limitations, arguments over inheritance versus composition, static versus dynamic typing,   and programmers arguing whether \"objects\" are data with behaviors attached or vice versa. So this sometimes puts me in an awkward situation when someone asks me what objects are since the myriad disagreements are strong, and valid.  I sort of punt on this, taking a \"meta view\" of what we're trying to accomplish when we write software. So I'll finish up by offering my view on objects, with an extended quote from my book Beginning Perl  .   ÆVAR THE PERSONAL SHOPPER  Many books have been written about OOP and even among experts, there is often disagreement about what OOP is. Many programmers have tried to explain OOP and leave the programmer confused. A case in point is the classic “An object is a data structure with behaviors attached to it.” Although that’s correct, that’s also an awful description and tells you almost nothing you need to know, so instead of giving you a textbook definition, we’re going to tell you a story.  You’re an awfully busy person and have little free time but plenty of disposable income, so you’ve decided to hire a personal shopper. His name is Ævar (any resemblance to reviewers of this book, living or dead, is purely coincidental) and he’s friendly, flamboyant, and most of all, cheap.  Because Ævar is new to both your city and the job, you have to tell him carefully how much money he can spend, exactly what quality of products you want, and where to buy them. You may even have to tell him which route to drive to pick up the goods and how to invoice you.  That, in essence, is procedural code and that’s what you’ve been doing up to now. You’ve been carefully telling the computer every step of the way what to do.  After a few months of explaining every little detail, Ævar gets upset and says, “þegiðu maður, ég veit alveg hvað ég er að gera” (Icelandic for “Shut up dude; I know what I’m doing”). And he does. He knows what you like and where to get it. He’s become an expert. In OO terms, you might now be doing this:  my $aevar = Shopper::Personal->new({ name => 'Ævar', budget => 100 }); $aevar->buy(@list_of_things_to_buy); my $invoice = $aevar->get_invoice;  You’re no longer telling Ævar every little step he needs to take to get your shopping done. He’s an expert, and he has all the knowledge needed to do your shopping for you and present you with the bill.   And that's it. Objects are simply experts. You tell them what you need and they get it done. Forget all of the handwaving about blueprints or \"data with behaviors.\" Those are implementation details. And once you start thinking about objects as simply experts about a particular problem domain, OOP becomes much easier.  ","title":"Alan Kay and OO Programming","url":"\/articles\/alan-kay-and-oo-programming.html"},{"body":" I may not win many friends by commenting on the emperor's nudity, but let's charge ahead anyway. I am not a clever man.  Possibly you've not heard of Douglas Hubbard . If you're interested at all in estimating the value of IT projects, his works should be required reading. In fact, his books are required reading for the Society of Actuaries if you want to pass their exams and become an actuary. Douglas Hubbard is a clever man.  But we'll come back to him in a moment.  If there's one key thing I've learned in consulting, it's that potential clients who start discussions by talking about building value for their customers have usually been better clients than those who start conversations with \"what's your rate?\" Why? Because the \"what's your rate\" clients are focused on controlling costs, not building value. I've found that their approach problems is different and those who are paying more attention to minimizing risk than maximizing value simply have different ideas about how to approach a project.  And you know what? I get it. If you want to go out to dinner and you only have £15 in your pocket, you're probably not going to a fancy restaurant. You just might grab a pint at a pub and hit a kebab shop after. Budget constraints are real.  But curiously, while development costs on a project are one of the first things we look at, they're also one of the worst metrics you can use to judge the value of a project. It's part of a problem that Hubbard refers to as the IT Measurement Inversion . Specifically:  The variables having the greatest impact on project success are the variables we measure the least.  So what sort of variables are there? There are the ongoing development and maintenance costs when the project is delivered. There are marketing costs. There are costs to have your people learn the new system. And so on. The costs are myriad and complex, but if we're going to judge the value of a new system, which variables matter the most? Those are the ones we should be focusing on.     Hubbard's work will guide you here. In particular, you might want to consider his book How to Measure Anything (that's not an affiliate link, by the way), where he teaches you how to break problems down to things that can be measured and how to measure them . You'll learn quite a bit about statistical analysis and running Monte Carlo simulations . In fact, one of the things you'll learn is that of the various techniques which have been developed to estimate project success, Monte Carlo simulations win hands down.  In fact, Monte Carlo simulations are so powerful that Hubbard talks about research at NASA showing that, for over 100 projects, the accountants outperformed the subject matter experts (scientists and engineers) on project estimates because the accountants used Monte Carlo simulations rather than other methodologies.  But for the purposes of our article, the key takeaway is that Hubbard has done a lot of this work for you and has nailed down what your real risks are. And it's almost intuitive once you think of it. Let's start with the most important.  The greatest risk to project success is whether or not it will be canceled.  That seems so brain-dead obvious that it's almost insulting to say, but when was the last time you quantified that risk? When was the last time you looked at the potential value of a large project and said \"there's a 25% that this project will be canceled\" and then factored that into your risk estimates? It's the single most important variable, but most people ignore it! Can you imagine running a company and discovering your managers are repeatedly ignoring the most valuable information at their disposal?  The next most important variable is system utilization. This is actually complex and includes how fast your product will roll out, whether people will use it, their rate of adoption, and so on. Again, this seems so obvious given 20\/20 hindsight. Sure, your project took six months to develop instead of three, but customers will be using it for many years. When it's put that way, it's obvious that estimating system utilization is more valuable than estimating development costs. But people still ignore it.  So let's look at development costs.  Of course you have to factor in development costs; I'd be insane to suggest otherwise. You have a budget and when you're looking at backing your project with Oracle or PostgreSQL, most of the time you find that Oracle isn't a terribly cost-effective solution. So yes, costs are important, but consider this: if the first version of Amazon's code cost twice as much to develop, Jeff Bezos would probably still be insanely rich. Uber would still be dominating their space. Facebook would still be selling your pesonal data to dodgy companies. I can confidently say that because these companies are providing real value to people (you might disagree about Facebook, but I can offer you 2.32 billion rebuttals ). The fact that these projects weren't canceled and are being used far, far outweighs the initial project costs. Or to restate all of this:  Initial development costs are incurred once. Your revenue stream is continuous. That's why development costs are less important.  \"But Curtis, I don't know how to do Monte Carlo simulations or estimate projected system utilization!\"  Well ... that's actually a fair point. These articles are intended, if nothing else, to be practical.  Since the two most importance variables in project success are the odds of it being canceled and system utilization, we need to reduce the risk of each of those. How do we do that?  Simply put, you go lean\/agile. Figure out the smallest possible work you can do which would add value and build that. Instead of a year-long project, you have a crazy one month race to get a simple prototype in front of your customers. If it turns out your customers have zero interest in your \"Recycled Food\" project, it's better to find out after a month instead of a year.  But let's say there's some interest. What now? Well, you've greatly reduced the chance of project cancelation because it looks viable, so you need to focus on system utilization. Amongst the key factors, getting customers to actually use the damned thing is incredibly important. You will be amazed at the ideas they come up with and the feedback they give you. As you push forward, you address customer concerns and have a lower risk of building features on top of things they say aren't working. You're constantly pushing new versions of the product in front of your customers and iterating based on their feedback. Every step of the way, you're improving your project based on customer behavior, not your hunches or market research.  This, incidentally, is why Lean Startups are such a powerful movement in business today. They assume they're not canceling their entire business, so they need to maximize system utilization. That means rolling the product out quickly and immediately responding to customer feedback. By doing this, assuming your product isn't canceled, you've now reduced the single greatest risk your project faces.  If you have any thoughts about this, please leave a comment below.  ","title":"Estimating Development Costs is Almost Useless","url":"\/articles\/estimating-development-costs-is-almost-useless.html"},{"body":"    Corinna in the Perl Core!  The Teams  Other Languages  What We’re Losing  You Can’t Violate Encapsulation  No Multiple Inheritance  No Inheriting from Blessed Classes  What We’re Gaining  The Principle of Parsimony  Conclusion     Corinna in the Perl Core!  It’s been a years-long, painful process, but with the release of Perl v.5.38,0  , the first bits of Corinna have been added to the Perl core. For those who have not been following along, Corinna is a project to add a new object system to the Perl core. Note that it’s not taking anything away from Perl; it’s adding a core object system for better memory consumption, performance, and elegance.  If you don’t have v5.38.0, you can still install Feature::Compat::Class  on Perl v5.14.0 and above and get the same features.  The MVP  for the Perl core is in seven stages, three of which are done and one of which (convenience attributes) is started (there is a :param attribute for field ).   Classes (done)  Inheritance (done)  Roles  Convenience attributes  Field initializer blocks (done)  MOP  Method modifiers   We need at least six of those to be implemented to really have something decent out of the box, but the MOP, while important, is not something that most OOP developers reach for on a regular basis.  You can read what’s currently implemented at perldoc perlclass  , but here’s a 2D point class using the existing features. We’ll make the default of the point be the graph origin.  class Point::2D { field $x :param = 0; field $y :param = 0; field $orig_x = $x; field $orig_y = $y; method x () { $x } method y () { $y } method reset () { $x = $orig_x; $y = $orig_y; } method move ($dx, $dy) { $x += $dx; $y += $dy; } method description () { return &quot;($x, $y)&quot;; } } my $point = Point::2D-&gt;new( x =&gt; 10, y =&gt; 20 ); $point-&gt;move( 5, 5 ); say $point-&gt;description; # (15, 25)  When the full Corinna is implemented, the method x and method y will be replaced with:  field $x :param :reader = 0; field $y :param :reader = 0;  The Teams  Here’s a partial list of the main contributors and\/or influencers to Corinna:   Damian Conway  Dan Book (Grinnz)  Graham Knop (haarg)  Harald Jörg  Matt Trout  Paul Evans  Sawyer X  Stevan Little   Note that Corinna was largely inspired by the work of Stevan Little. Damian Conway was also a huge contributor, but he was mostly behind the scenes sharing email with me and posting to blogs.perl.org  ;  Sawyer X’s main contribution was being the pumpking who encouraged me to stop trying to implement Corinna, but instead focusing on building something great without worrying about the current limitations of Perl.  Paul Evans is doing the actual implementation work and he’s doing a fantastic job.  The others (not to slight them, I can’t remember all the details of a multi-year project) were also huge contributors. They helped suss out design flaws, called me on my bullshit, and helped me understand the limitations of what we were trying to accomplish.  It’s also worth noting that due to this team effort, Corinna is not what I wanted. I wanted more. Much more. But it wasn’t just about trimming Corinna down to what P5P would accept, it was also vetoing many of my ideas about what I thought should be in Corinna. I was wrong about a lot of things and I’m sure I’m right about about some others, but the team effort is what made Corinna what it is today and I’m immensely grateful for that.  Other Languages  In the process of developing the Corinna MVP, many other languages were looked at. I was particularly curious about Beta  , but there was nothing that fit. I also looked at Smalltalk, Java, Raku, Python, Eiffel, and Ruby (and probably others I’m forgetting right now). There are many interesting ideas, but given that we were trying to get something that fit Perl, the biggest influences were Raku, Moo\/se, and yes, Java. Primarily, I was interested in protected methods and inner classes, but the former seemed problematic and the latter wasn’t something appropriate for the MVP. However, reading about OOP design (so many examples are in Java), helped quite a bit with Corinna.  What We’re Losing  There are, of course, trade-offs in something this huge.  You Can’t Violate Encapsulation  The biggest concern amongst developers is that we can no longer violate encapsulation (well, you can via the MOP, but it will be more work, as it should be. Emergency patches sometimes require violating encapsulation).  However, I don’t know many developers who can define encapsulation. People keep saying it’s about hiding state, but that’s not true. It’s about hiding state and process because the two are intimately linked in classes. If you can change the state without going through the interface, you have no guarantees the process is correct. That’s a bad thing.  Fortunately, this has been well-understood for decades, so there’s a wealth of information about proper OOP design on how handle this. I expect there will be changes to Corinna to help with this, but we don’t know what they are yet.  No Multiple Inheritance  To my surprise, no one on the design team strongly argued against this change. There were discussions, but nothing too serious. There are many OOP languages which do not allow multiple inheritance and it’s not a big deal. In fact, there are some OOP language which don’t allow inheritance at all.  Most languages which forbid multiple inheritance offer some sort of alternative. In our case, it’s roles.  No Inheriting from Blessed Classes  Your new Corinna class cannot inherit from anything not declared as a class . This was a slightly contentious issue, but there are simply too many differences. Corinna doesn’t allow you to “reach inside” and manipulate the data. Corinna is single-inheritance, so it’s unclear how that would have worked with classes allowing multiple inheritance. And ultimately, we’d like Corinna to have the option of having a different UNIVERSAL:: base class, offering different capabilities.  This, of course, means that developers will have to understand “composition over inheritance.” This is a good thing.  What We’re Gaining  Object-oriented programming will be “first class” in Perl, not just a hacked on solution. Methods and subroutines will no longer be the same thing. We’ll have private methods. We already have a much more concise, and clear way of writing classes. Companies using the Object::Pad  prototype have reported a 10% reduction in CPU usage and that’s without optimization. We have have potential for a much more efficient implementation of objects than we’ve had before.  The Principle of Parsimony  Given all of the above, I have confession to make about the new class syntax for Perl: we made a mistake. It’s a whopping huge mistake.  The problem is that we don’t know what that mistake is. There’s probably more than one of them.   In the README of the Corinna repository is a section entitled The Principle of Parsimony  , it states:   Many things in the proposal are deliberately restrictive, such as Corinna only allowing single inheritance. This is to allow Corinna to be cautious in not promising too much. If we later find this too restrictive, we can allow multiple inheritance. However, if we start with multiple inheritance and discover we don’t need or want multiple inheritance, we would break existing code by taking it away.  Any proposals to change the RFC must consider the principle of parsimony.   The design team really should have highlighted this more because it’s the single most important thing (outside of the new class syntax itself).   Most people seem happy with the changes. They’re new. They’re exciting. They’re a step towards a new future for the language.  Some people are unhappy. Some are unhappy for reasons that are, well, silly. Case in point: I’ve had a developer angrily tell me they’re nearing retirement and they don’t want to keep up with a bunch of changes in the Perl language (more than one, but the other wasn’t quite so strident in this position).  Some are unhappy because Corinna didn’t go far enough.  Some are unhappy because Corinna has gone too far.  Others are unhappy because « insert reason here » .  I get that. I really do.  So why doesn’t Corinna have multiple inheritance? Because if it did and it’s a mistake, we can’t easily walk it back without breaking existing code.  So why doesn’t Corinna allow inheriting from non-Corinna objects? Because if it did and it’s a mistake, we can’t easily walk it back without breaking existing code.  So why doesn’t Corinna have native support for field (attribute) builders like Moo\/se? Because if it did and it’s a mistake, we can’t easily walk it back without breaking existing code.  The counter-argument is that the new class syntax is experimental and we should have that freedom. My counter-counter-argument is that I know of large code-bases which haven’t upgraded past 5.26 because 5.28 swapped the order of attributes and signatures bodies  .  Companies can and do rely on experimental features.  The design team agreed (generally) to follow the Principle of Parsimony to ensure we didn’t paint ourselves into corners more than we already had. We expect that in the real-world, people are going to make compelling arguments for changes based on limitations they discover. We know mistakes are going to be found. We’d like to see if those are mistakes in our design, or mistakes in how people (ab)use OOP code in Perl.  I might add that the Principle of Parsimony hasn’t been an absolute, even in the design stage. For example, originally Corinna required a postfix block:  class Foo { ... }  The extremely tight scoping of the postfix block would make it much harder for Corinna keywords to bleed into an unwanted scope. I argued for that, but I was overruled. The postfix block is not required.  So please keep this in mind about future discussions of the new class syntax.  We made mistakes. We don’t know what they are. We worked very hard to try to ensure that fixing those mistakes means extending Corinna, not breaking it.  Conclusion  Corinna, for now, is experimental and incomplete. The biggest obstacle for its adoption is likely to be the fact that developers are going to have to learn new ways of programming. The most annoying might be that field variables aren’t available in subclasses (encapsulation!), so you’ll find it convenient to make a public accessor to fetch this data. There’s a similar issue with roles. The Java notion of protected methods is not available.  There are ways we could potentially address these issues, but we need to see what people are doing with Corinna and what their real pain points are.  Fortunately, the Object::Pad project shows that the new approach plays quite well with existing code, so we are pretty sure we have the basics right, but we don’t know what we don’t know.  Interesting times are ahead.  ","title":"Corinna in the Perl Core","url":"\/articles\/corinna-in-the-perl-core.html"},{"body":" A fellow agile trainer recently told me that many of his customers are fleeing scrum to get away from the zealots. I sighed and nodded in agreement—all too familiar with those agile enthusiasts who repeat the \"agile isn't a silver bullet\" mantra—and then proceed to recommend it for everything.  Most would agree that not everything must be agile, but when you ask for examples of what the agile methodology isn't good for, you get blank stares. To figure out when you should and shouldn't use agile, you need to understand how products evolve in the market, determine where your product fits on that continuum, and reexamine agile in that light.  The four phases of product evolution  Products tend to evolve through four maturity phases. Each has unique characteristics and should be treated differently. They include:   Novelty: This is the \"gee whiz\" phase, where a new and exciting idea emerges and people begin to wonder what can be done with it. For example, when online shopping first emerged, people wondered if you could sell clothes, food, or cars online. Would people trust you with their credit card information? Was there really a market for people buying goods from faceless, anonymous entities online? We know the answers now, but we didn't then.  Stability: This is the phase when people start to understand what they can do with a given product. The focus shifts from what can be done to how to do it. Remember all those \"how to build a shopping cart\" books and articles in the late '90s and early '00s? This was online shopping transitioning from novelty to stability.  Commodity: This phase kicks in when the market has stabilized enough that for most companies, building their own version of a product doesn't make economic sense. Instead, there are enough competing products out there that acquiring the product off-the-shelf is simpler and cheaper.  Utility: This is where you no longer purchase a product, but you consume it as needed, just as you would with electricity. Shopify, Etsy, and many other e-commerce utility solutions are readily available.   Of these phases, the novelty and utility phases are the most interesting. What makes the novelty phase interesting is obvious—startups make fortunes in this area. However, the utility phase is also fascinating if you're the one providing the utility. Products in the utility phase tend to be high-volume, low-profit systems. Entire industries are built on them. Electricity started out as a novelty and today is a well-established utility with industries such as television, radio, and computing built on top of it. Cars are going through this phase: with the rise of Uber, Lyft, and self-driving cars, the market is facing the prospect of cars becoming utilities. Someone is going to make billions when they figure out how to exploit the new markets built on top of this change.  One characteristic of the novelty phase is the rapid change and uncertainty. This doesn't exist for the utility phase—customers will be reluctant to consume your utility if rapid change and uncertainty is another price they have to pay.  Agile's job: To reduce the cost of change and uncertainty  Once you understand where a product is in its evolution, the appropriate use of agile should become clear. Agile is nothing more than a method of reducing the cost of change and uncertainty.  This is reinforced by the fourth point in the Agile Manifesto: \"responding to change over following a plan.\" In fact, while many have read the Agile Manifesto, few have read the twelve Agile Principles associated with it. Many of the principles reinforce the point that agile is about embracing change and lowering cost. If you think agile is all about standups, short iterations, retrospectives, story cards, and so on, then you've missed the point. You're not alone: many practitioners have been misguided because agile is often taught as a set of behaviors, with little reference to the goal beyond vague mentions of side-effects like improved time to market, customer satisfaction, and lower startup costs. It's like teaching people the moves of chess and not teaching them about checkmate. In this light, scrum, Extreme Programming (XP), Kanban, and other methodologies are tools you use after you've already decided on an agile approach.  So when should you choose agile? When you understand both agile and product evolution, the answer is simple: the newer the product and its market, the better fit with agile.  Agile works best in environments with the expectation of change, when you need to optimize for uncertainty. Products in the novelty phase definitely benefit. However, what if you pretty much do know everything up front? What if there is little change and uncertainty in your environment? One agile mantra is to \"fail fast.\" Do you really want to fail fast if you're building a nuclear reactor? Of course not. Nuclear reactors are one attempt to implement the utility of electricity. Agile isn't scrum, XP, or Kanban. Agile is merely a way of reducing the cost of uncertainty and change. If your project is close to the novelty phase of its market evolution, you'll have a lot of uncertainty and change. In this situation, agile (reducing the cost of change) is a great choice. But if your product falls into the stability and commodity phases, consider switching to a lean methodology (minimum viable feature set with rapid development, reducing the cost of waste). Finally, if your products are commodities that are perhaps evolving into utilities, consider techniques such as Six Sigma (reducing the cost of inconsistency). Why? Because they focus on reliability and reproducibility. Optimizing to lower the cost of change and uncertainty doesn't make sense when you already have little change or uncertainty. Not for everyone  Many agile practitioners object at this point. They insist that you can simultaneously optimize for change, uncertainty, reliability, and reproducibility. For those people, here's a thought experiment. Imagine you've decided to create a tool to allow job applicants and companies to rate recruiters. Using crowdsourcing, you can start to identify the best recruiters for your needs. Put five excellent, independent agile teams on this project and you'll have five different products at the end. Reproducibility simply isn't something agile was designed for.  Don't get me wrong: I'm a huge fan of agile. I've happily embraced it for years, and I enjoy working in agile environments. But it's time we end the tyranny of agile and recognize that it's merely another tool in the toolbox. Agile is a great hammer, but not everything is a nail.  Note: I originally published this article on TechBecon .  ","title":"When to Choose Agile","url":"\/articles\/when-to-choose-agile.html"},{"body":" My discussion of Alan Kay's ideas behind OO programming made the rounds, getting about 13,000 page views (as of this writing) from various sources, including Hacker News  and Reddit  . There was quite a bit of discussion, most of which I'll cheerfully let people decide for themselves, but there's one perennial question, as exemplified by this response  , which I really need to address.  So, what exactly should happen when you request last_name on an object that doesn't respond to that message? ... It tells me you haven't been the one receiving those 4AM phone calls, angry dictates from C-levels, and responsible for hundreds of thousands of dollars (or even millions) of lost revenue per minute of down-time.  There was a lot of vitriol in the comment and I didn't want to respond to that, but in this case, that vitriol actually strikes to the heart of this issue. Specifically, I have been on pager duty for several companies, getting those frantic early morning phone calls and that's helped to guide my thoughts on what robust computing should be, and how you might handle objects that don't respond to messages.    When you don't handle messages gracefully. Source   Back in the 90s, when I was a COBOL programmer for a huge insurance company, I was responsible for writing software that would integrate third-party data into our accounting system. Whenever I would write new software, I would send that to our \"operators\", along with a series of steps on how to restart the software if any step in our JCL  failed. I really didn't want to get a late-night call, so I tried very hard to ensure those instructions were clear, and even took the naughty step of allocating more disk space than my software needed to guarantee that this would never be a problem with my software. I knew the main accounting software ran out of disk space every couple of weeks, requiring costly work on the part of the developers and operators to coordinate on resurrecting a batch job.    Obviously, that was much earlier in my programming career. I didn't know anything about OOP, had never heard of Alan Kay, and couldn't possibly understand the core issues he was discussing, but I had done one thing right: I wrote down very, very complete documentation that intelligent people could use to restart my software if it failed. Keep that in mind: I gave a lot of thought about how to respond if my software didn't perform as expected.  Fast forward a couple of decades to when I was hired to fix the ETL system of a company working to reduce the costs of Phase III clinical trials.  .   It was a large project, with many parts, but the ETL system had been written a couple of years earlier by a doctor! I was rather surprised. It was clear that he wasn't an experienced developer, but his work showed a lot of brilliance because he deeply understood the problem space.  The ETL system was a bottleneck for this startup and honestly, no one wanted to touch it. A new dataset would arrive from a pharmaceutical company and it would take two days of manual effort from a programmer to get the dataset into a shape where it could be run through the \"ETL\" system. I say \"ETL\" in scare quotes because obviously you don't want to take a highly-paid software developer off of their work for several days of manual data cleanup just so you can process an Excel document. But this work was critical to the startup and while they dreaded getting new data, they had to handle it. So they offered me a contract and the other developers wished me luck, happy to be washing their hands of the mess.  After a few months, I had rewritten the ETL system to be a fault-tolerant, rules-driven system that could handle the radically different data formats that pharmaceutical companies kept sending us.   More importatnly, instead of two days of hands-on work from one of our developers, it turned into two hours of automated work, leaving the developers free to do other tasks. So given that the data quality was rubbish, how did I get there?  I had to jump through a lot of strange hoops, but here's an interesting key: the pharmaceutical companies were astonished that we could get a year's worth of work done in a month. They simply were not able to change their processes and asking them to \"fix\" the data they sent us failed. So my job was to fix it for them. And that brings me back to the main issue with Alan Kay's idea about messaging. What do you do if you send a message and don't get a response? Here's how we programmers tend to think of that:  try { \/\/ This rubbish method names helps to illustrate that we \/\/ might have an unreliable source result = someObject.fetchDataFromServer(orderId); } catch(UnknownOrder uo) { System.out.println(\"Unknown order id: \" + uo); } catch(ExistentialCrisis ex) { System.out.println(\"Server is sitting in the corner, crying.\"); } catch(Exception e) { System.out.println(\"Yeah, this blew chunks.\"); e.printStackTrace(new java.io.PrintStream(outputStream)); }  This is the software equivalent of \"Not My Problem.\" After all, the fetchDataFromServer() method should return data! It's not my fault if the other team's server sucks!  But I had no way of telling a bunch of multi-billion dollar pharmaceutical companies to get their act together. Making the ETL system work very much was my problem. And by taking responsibility for that problem, I wrote a much more robust system. So what do you when fetchDataFromServer() fails? You do the same thing I did at the insurance company when I sent instructions to the operators explaining how to \"fix\" things when the software failed. Except that I'm sending that to my software, telling it how to fix itself. This requires teaching developers a different way of thinking about software, something that goes beyond try\/catch , \"not my problem.\"  So when fetchDataFromServer() fails, or when an object doesn't respond to the message you sent, what do you? There's a simple way to approach this: ask yourself what you would do in real life. What if you deleted your partner's phone number by accident and you need to call him. Do you just give up and say \"not my problem?\" Of course not. If he's ever sent you a text, you probably have his number on the text. Maybe he sent you an email containing his number. Maybe your friend has his number and you can ask them. Or maybe you can find a different way of contacting your partner. The choice is yours: give up, or take responsibility for finding a solution.  Hell, maybe we should call this RDD, \"responsibility-driven development,\" because what the programming world needs is new acronyms.  For the ETL system, I would have problems such as needing to know if \"Dr. Bob Smith at Mercy Clinic\" was the same person as \"Robert Smith (no title) at Mercy Hospital.\" Sometimes those different records would come in from different pharmaceutical companies, but sometimes they would come in from the same pharmaceutical company; I'd still have to figure out if they were the same person. This is the very definition of \"bad data\" but it was critical that the ETL system work and, as it turns out, it's a fairly well-understood problem and there are many approaches.  In this case, I would use various geolocation services to find addresses and lat\/long data for the locations (with fallbacks if a service was down). I would create known \"name variants\" (in English, \"Robert\" is often shortened to \"Bob\"), encode names with something like Soundex   , stripped stop words from names, check email addresses and phone numbers, along with their Levenshtein distance since they were a frequent source of typos, and so on. By the time I was done, I had a large list of things which were potentially identifying, and scoring to indicate which sources of \"truth\" were more valuable.  By calculating a score, if it exceeded a particular threshold, we assumed they were the same person. Below a threshold, we assumed they were different people. Between those thresholds, we flagged them for follow-up. And if the external geolocation services were down? Fine! I just didn't get a score for their data, but I still had plenty of other scores to help me calculate that threshold.   Surprisingly, we usually had no follow-up records, despite often processing tens of thousands of people per batch.  This is how you deal with unanswered messages. You think about your response instead of just letting your software crash. Is it OK if you don't have that info? Can you just leave a spreadsheet field blank? Are there other sources for that information? Can you try later to fetch that information, having designed a nice async system to make this easier? There are tons of ways of dealing with these issues, but they all involve more work, and different problems, of course, involve different solutions.  This isn't to say that there's always a neat answer, but curiously, as I find problem spaces increasing in complexity, I sometimes find that there are more solutions to bad data instead of just \"more bad data.\"  There is, however, a little bug in this ointment: clients often want software today, not tomorrow. As we're pushed harder to develop more software in less time, struggling under the tyranny of budgets , we often don't have the time to make our software this robust. Even if it would save the client a lot of money in the long run, they want to move quickly and let their later selves pay down the technical debt. So I'll be writing a more try\/catch blocks in the future, because sometimes it's really not my problem.  ","title":"Alan Kay and Missing Messages","url":"\/articles\/alan-kay-and-missing-messages-a-follow-up.html"},{"body":" Note: The Model, View, Controller pattern, or \"MVC\", actually comes in many flavors, but for the purposes of this article, I'm referring to MVC for the Web. Feel free to comment below, but keep in mind that we have a particular viewpoint here.  The Problem: getting MVC wrong is expensive and costs you a lot of money.  I often start working with a new client as a manager or developer and find a Web app with a common set of mistakes . On one hand, I would like to see software developers learn from past mistakes. On the other hand, this pays my bills. In particular, MVC&mdash; which has unfairly been getting a bad rap &mdash;is thoroughly misunderstood and much of this can be blamed on rubbish examples and tutorials.  Here's a perfect \"bad\" example from an MVC explanation using Python and Flask :  @app.route('\/') def main_page(): \"\"\"Searches the database for entries, then displays them.\"\"\" db = get_db() cur = db.execute('select * from entries order by id desc') entries = cur.fetchall() return render_template('index.html', entries=entries)  In this example, we have a page which finds \"entries\" and displays them on a Web page. It's entirely possible that you see no problem with this, but it perfectly demonstrates what's wrong with people's understanding of MVC: you've pushed knowledge of your database, your business logic, and your HTML into the controller.  That's what our examples and tutorials are teaching developers. Because of this, when I work with a new client using \"MVC\", what I really see is this:  def some_controller_method(): \"\"\"begin hundreds of lines of code\"\"\"  How do I hate thee? Let me count the ways. (with apologies to Elizabeth Barrett Browning ). The following comes from years of experience seeing this over and over again.   The database or ORM is exposed at the controller level (nooooo!!!!!)  You can't reuse that logic outside of the controller  You probably can't test that logic outside of a Web context  Tests are much slower because of the web context  Test coverage suffers because it's hard to test conditional logic in the controller  Many parts of that logic get duplicated in other controller methods  Different behaviors become tightly coupled, making it hard to develop  Performance suffers because you can't cleanly target a layer  Integrating new technologies is harder because the layers are tightly coupled  ... and so on   But what does this mean in practice? Let's consider a real example I encountered a few years ago.  One company I worked with had all of the above problems. In particular, they had a 3,000 line \"method\" for searching for products. It conflated many things, including searching for products and filtering the search results. Thus, even though I was tasked with working on it, every time I touched search, it would break filtering. Every time I touched filtering, it would break search. It was slow and tedious to get anything done. It took me some time to separate search and filtering and when I was done, I had a surprise visit from one of the developers of the mobile side. It seems they wanted a proper search engine for ages but never added it because they couldn't separate the searching (which they needed) from the filtering (which they didn't). They thanked me because, after my work, it took them 15 minutes to implement a search engine.  Not getting this right flushes money down the toilet . Often we hear that a new feature will take a lot of time to implement \"because the application wasn't designed for this use case\" when what is really meant is \"because we didn't understand separation of concerns and thus a one week feature will take months.\"  Note:  you can fix this mess, but it takes time and you have to understand what MVC should look like.  A Proper Controller Method  To contrast this, let's look at a the controller for an extremely complex bit of logic from the free MMORPG, Tau Station , using Perl and the Catalyst MVC framework :  sub station ( $self, $c, $station ) : Local Args(1) { $c-&gt;character-&gt;travel_to_station($station); }  Without going into detail about what travel_to_station does (trust me, it's complex), all we see is a request for a particularly complex action to be performed. In the controller we can see:   No knowledge of how the model is constructed  No knowledge of how the data will be used  Knowledge of the route to this controller method (the Local attribute)   Having the route information hard-coded is a bit unfortunate, but that's easy to work around. However, this controller method merely connects the model to the view, and that's all. That's what you're looking for in MVC. MVC, when appropriate, takes an application with a UI (user interface), and decouples the responsibilities into reasonable layers.  It's now easy to test the logic (you're testing the model, not the controller layer and all of its web context). It's easy to reuse this code. And, if the model is properly built, you have a great separation of concerns that makes it easy to extend, maintain, and repair the application.   Model  The application.  View  What the consumer (user) sees. Often HTML, JSON, or XML.  Controller  A thin layer connecting the View and the Model.   And that's it! There's nothing fancy, but it often requires explanation. The controller, as shown above, should be as thin as possible. The view should only have display logic and the model is ... what, exactly?  The Model  I'll write more about this another time, but the model is the part where everything breaks down. Almost every MVC tutorial that I see confuses the data model with the Model. This is what we get (pseudocode):  method neighbors(): model = this.get_orm(); neighbors = model.table('Characters') .search( location: this.character.location ); this.template(\"people\", people=neighbors);  Now that looks reasonable. In fact, it seems so sensible that I did something very similar in the Tau Station MMORPG a long time ago. But it was deeply flawed. It turns out there are many cases where you need to know the other characters in your vicinity and this logic should be encapsulated. But it's just one method call, so factoring it out is silly, right?  Wrong. As it turns out, you didn't want to show any characters who hadn't been active in the past day. Well, unless they were NPCs. Or the other characters have something else which makes them \"non-visible\". Any time we discover a new case of when characters are or are not visible, we can visit this logic in every controller method, or we can abstract it out into a single method:  So let's try it again:  method neighbors(): model = this.get_orm(); neighbors = model.table('Characters').visible_to(this.character); this.template(\"people\", people=neighbors);  Ah, that's much better. Except we have exposed our ORM (and thus, our database). We have many data sources which have data in Redis, or configuration files, or a custom cache, or from all sorts of locations that the controller should never, ever know about. Some of my clients with this anti-pattern have worked around this by trying to implant those other data sources directly in their ORM, which simply shuffles the coupling around. The controller should never care about the source of data. The controller should only be asking for the data. Instead:  method neighbors(): neighbors = this.model('Characters').visible_to(this.character); this.template(\"people\", people=neighbors);  Where do the \"visible\" characters come from? You don't care. By hiding all of these details from the controller, the developers are free to implement the fetching of this data any way they want, so long as they respect the contract originally specified when this method was created.  The model is your business logic. And I'll go a step further and say a good design choice is to make the model provide services the client needs, with a headless application underneath. By creating a this, you can layer a controller and HTML view on it. If you're careful, you can have your controller target multiple views and build a native mobile application on top of it, only requiring a new view. You could create a native client\/server desktop application, too. If the controllers are too inflexible, that's OK. They're tiny and easy to skip. Write thin controllers specifically for your mobile and desktop apps, secure in the knowledge that the application logic is all in your headless application.  There any many benefits to this architecture and I'll cover them in a later article. Further, the headless application needs its own architecture. When is the last time you saw an MVC explanation go into detail about the architecture of the model? That's a rare beast indeed and it's why so many companies have themselves mired in technical debt: they get a partial explanation of a complex concept and start building from there..  ","title":"Fixing MVC in Web Applications","url":"\/articles\/fixing-mvc-in-web-applications.html"},{"body":"    Introduction  Inheritance  Classes versus Prototypes  Dr. Alan Kay  Messaging  Isolation  Extreme Late Binding  Microservices  A Little Secret     Introduction  I’ve been meaning to write this article for a while, but never quite got around to it. Now I’m around it and I know that it will annoy a few people. In fact, any time I read new discussions about how Dr. Alan Kay, the inventor of the term “object-oriented programming”, intended something entirely different from what we have today, the online discussions often devolve to people acting like he’s some grumpy old dude who just graduated from a PHP bootcamp.   If you want some extended discussions I’ve written about Alan Kay and OO Programming and Alan Kay and Missing Messages . They’re not required for following along in this article.   In 1967, Simula 67 was released. This is often referred to as the first “true” object-oriented language because it included these four feature:   Classes  Polymorphism  Encapsulation  Inheritance   It offered many other features, such as subtyping, coroutines, and discrete-event simulation  , but those are just curiosities for the purpose of this article.  Inheritance  Of the above list, the first three are generally non-controversial, but inheritance has been a pain in the ass for over five decades now. The following illustrates the core problem with inheritance:  class Person :isa(Invoice) { ... }  What the hell does that even mean? I suspect (though I can’t prove), that the majority of issues we have with inheritance today stem from that core problem. A subclass is intended to be a more specific instance of a parent class, but that’s a semantic issue the programming language can’t determine. Instead, there’s a kilo of fat shoved in our skull which has to figure out what that means and we often get it wrong.  This is problematic enough that Alan Kay left inheritance out of his original descriptions of Object-Oriented Programming (OOP). It’s problematic enough that some OOP langauges (or OOP-like, depending on your definition) don’t allow inheritance at all. Some allow multiple inheritance. Others only allow single-inheritance but usually give you some tools to make life a touch easier than simply composition and delegation (interfaces, mixins, aspect-oriented programming, roles (Smalltalk-style traits), etc.).  Importantly, in class-based OOP languages, the classes define the behavior and the instances of those classes contain the data (hand-waving on this quite a bit, of course).  Classes versus Prototypes  But what about the other features? They seem good, right? Well, there are a number of protype based programming langauges  , such as Javascript, Lua, Self, Io, and so on. In these languages, an object can contain both data and behavior and you copy the object and add new data and behavior as needed. To be honest, though, when I squint, I see single inheritance in protype-based languages. I’ve never worked with one enough to be confident in saying that, though.  Dr. Alan Kay  But here’s someone who should be considered an expert on the term. Note that he didn’t invent OOP, merely the term. And he’s not happy with where it went.     This isn’t a classical “argument from authority” fallacy because I’m not going to argue whether or not he’s correct. Instead, I’m going to argue that microservices adhere much more closely to his idea of OOP than traditional class-based systems.  In particular, he identifies the following three key features as the core of his idea of OOP  :   Messaging  Isolation  Extreme late-binding   All three of those are unusual enough to the minds of many programmers that they’re worth digging into. I’ll be brief.  Messaging  In Kay’s view, you don’t call methods on an object, you send messages to it. The receiver is free to respond however it likes, including ignoring the message. For example, with a web server, you could send the message DELETE \/foo.html and it’s free to return a 400, a 404, another status code, or simply drop the request. If you think of the server as an object , you’ve sent it a message and it can do any damned thing it wants with it.  Isolation  Alan Kay actually said, “local retention and protection and hiding of state-process,” but we often use the term “isolation” instead.  First, let’s think about encapsulation. If, internally, an object has a secret key for encrypting\/decrypting something, you don’t want that exposed. That’s encapsulation . Many languages get this right, though I’ve found that many dynamic languages make this hard to enforce. Fixing this is one of the key drivers of the Corinna OOP project for Perl  .  But what happens if we do this (Corinna OOP syntax)?  class User { has $username :param :reader; has $password :param; } my $user = User-&gt;new( username =&gt; &quot;Ovid&quot;, password =&gt; 'hunter2', ); say $user-&gt;username; # &quot;Ovid&quot; say $user-&gt;password; # &quot;Method not found&quot; error  So I can instantiate a User object, send it to you, and you can see their username, but not their password. That’s encapsulation. But if you try to call $user-&gt;password , you also get an exception. That’s encapsulation, but not isolation. Alan Kay referred to “state-process”, not just state.  Again, think about a web server and a web client. If the server throws a 500 error, or just catastrophically crashes, your client doesn’t suddenly crash (unless it’s poorly written). Instead, it gives you useful information which you can then use to decide what action to take. In fact, Kay has described the Web as being the most successful example of OOP ever invented.  Extreme Late Binding  In many OO languages, we have early binding where the class is used to resolve, at compile time, the method that is to be called. You will get compile time errors if the method does not exist.  In late binding, we use the instance to determine resolve, at runtime, the method that is to be called. If you don’t know what kind of instance you have, you can get runtime errors if the method does not exist.  So what’s extreme late binding? The method might not even exist when it’s called, but it still resolves, or its behavior might change between calls because you’ve found a more efficient or accurate implementation! Smalltalk and Erlang are two languages known to easily support this. Surprisingly, Perl can as well, using the AUTOLOAD feature. Here’s a silly example using the Corinna syntax, with a few speculative (made-up) features to make the example clearer:  class SomeClass { has @objects; ADJUST { # populate @objects } method AUTOLOAD ($method, @args) { foreach my $object (@objects) { if ( $object-&gt;can($method, @args) ) { $MOP-&gt;class-&gt;add_method( $method, method { $object-&gt;$method(@args) } ); return $self-&gt;$method(@args); } } return; } } my $instance = SomeClass-&gt;new; # return $object-&gt;frobnicate(42) from the first composed object # which can handle the request $instance-&gt;frobnicate(42);  In the above example, SomeClass doesn’t implement any methods at all, but if you call one that doesn’t exist, it installs one that delegates to an object having that method. Later calls would call the installed method instead of AUTOLOAD .  We can again use the web server\/client analogy to explain this. A browser makes a request and the web server may not even be designed to handle the request but serves a result anyway. It might even later cache that request so it doesn’t have to recompute it. You don’t know. You don’t care. You only know you made a request and got a response. Hopefully it’s a useful response.  The intent behind extreme late-binding is to protect you from committing too early to an implementation that may not be what you need (or even needed), but allow you to respond to the method\/message if appropriate.  Microservices  There’s a lot of confusion about microservices and SOA (Service-Oriented Architecture) and I cover some of it in this article  .   Microservices are generally expected to have all the logic and data they need to perform a task. They’re not coordinating with a bunch of other services and thus are loosely coupled and fit well with agile development.  SOA services, however, need to coordinate with other services in the same way that your Order object needs a Collection of Item objects, along with a Customer object, and so on.   A microservice is self-contained and might duplicate code in a way that SOA does not, but it’s completely isolated from the rest of the code. It’s basically a process\/server running that has an API (messages), isolates everything , and can use extreme late-binding if it wishes to.  Thus, if you aspire to Dr. Kay’s model (and the success of the internet is a powerful testimonial), you may want to look at microservices. Some people look at microservices as being a silly fad, but they’re a fad that’s been around for quite some time and more and more companies are using them. In fact, our company, All Around the World  is routinely getting more requests from clients about implementing SOA\/microservices for their systems.  One interesting point about microservices (which SOA might struggle with) is that the extreme isolation of them combined with an OpenAPI interface means that you can separate off a layer of your system as a microservice, make sure it works, and then easily rewrite it in another programming language that might be more appropriate to the task at hand. That’s often extremely hard to do when you directly use\/import\/require\/include a module because it’s usually required to be in the same language.  A Little Secret  But if you agree with the idea that microservices might be the next step in OOP, along the lines of what Alan Kay proposed, perhaps we should keep this our little secret, shall we? OOP “purists” might be horrified at the description and OOP “haters” might be aghast at realizing they’ve been writing objects all this time. Admitting that microservices are objects might be a marketing disaster, even if it’s true.  ","title":"Are Microservices the Next Phase of Object-Oriented Programming?","url":"\/articles\/are-microservices-the-next-phase-of-object-oriented-programming.html"},{"body":"     We say what we do and do what we say.    Note: I’ve written about this before, but the writing is scattered over the web and in conference keynote presentations I’ve given. This is to bring it home to one site. This code written by All Around the World  is driving the narrative sci-fi MMORPG Tau Station  . And if you feel the need, feel free to email us to discuss how we can build a remote software team for you. This is the quality we have been delivering for years, both for Tau Station and for our clients and therefore the quality you can expect for your own project.  Just about anyone who’s taken a beginning database course learns about the critical importance of database transactions. For example, consider the following code:  sub transfer_money ($self, $from, $to, $amount) { if ( $from-&gt;balance &lt; $amount ) { Exception::InsufficientFunds-&gt;throw(...); } $from-&gt;dec_balance($amount); $to-&gt;inc_balance($amount); }  The code is a grotesque oversimplification in several ways, but let’s just look at one little line:  $from-&gt;dec_balance($amount);  What happens if that fails and throws an exception? If it fails before the money is withdrawn, that might not be disastrous. If it happens after the money is withdrawn, we withdraw the money, but don’t deposit it to the new account. Our customers will be disappointed, to put it mildly.  So we fix that with a database transaction. It might look like this:  sub transfer_money ($self, $from, $to, $amount) { if ( $from-&gt;balance &lt; $amount ) { Exception::InsufficientFunds-&gt;throw(...); } $self-&gt;db-&gt;do_transaction(sub { $from-&gt;dec_balance($amount); $to-&gt;inc_balance($amount); }); }  Now if dec_balance fails, we’re guaranteed that whatever money was withdrawn will still be in the original account.  But it still has a massive bug. What if money was withdrawn after we checked the balance, but before we withdrew it? Oops. It’s called a “race condition”, because the behavior of your code depends on which code finishes first.  In this case, the behavior depends on whether the system allows negative balances, whether the account allows negative balances, and other factors. We don’t want to risk this bug, so let’s expand the transaction scope.  sub transfer_money ($self, $from, $to, $amount) { $self-&gt;db-&gt;do_transaction(sub { if ( $from-&gt;balance &lt; $amount ) { Exception::InsufficientFunds-&gt;throw(...); } $from-&gt;dec_balance($amount); $to-&gt;inc_balance($amount); }); }  So now we check the transaction before we check the balance. Good to go, right?  No. Of course not. For most OO system I’ve worked on, that code is not good to go because objects hold their data independently from the underlying storage mechanism. What that means is that the information was read earlier and might be stale. Instead, you need to refresh your data. But you can’t just do a SELECT , you need a SELECT ... FOR UPDATE to ensure that the row is locked in the transaction.  sub transfer_money ($self, $from, $to, $amount) { $self-&gt;db-&gt;do_transaction(sub { $from-&gt;requery_for_update; $to-&gt;requery_for_update; if ( $from-&gt;balance &lt; $amount ) { Exception::InsufficientFunds-&gt;throw(...); } $from-&gt;dec_balance($amount); $to-&gt;inc_balance($amount); }); }  The above is certainly better, but my goodness, there are traps there. And it still has bugs, but I’ll stop belaboring the point now.  All you wanted to do was move a bit of money from one bank account to another. And honestly, this is a simple example. When your code gets seriously complex, it can be hard to track down bugs caused by race conditions and not having proper scope on transactions.  Which brings me to the real point of this article:   The more things you must remember, the more things you will forget.   That is something that developers often gloss over. “You have to know your system.” “It’s your fault that you weren’t paying attention.” “Be better!”  No. You want to write systems that take failure modes into account and make it hard to write those serious bugs.  Making This Simple  So, what would the above look like in Tau Station  ? Well, currently we don’t have multiple bank accounts per player, but we do have the ability to move money from your wallet to your bank account and conceptually that’s the same thing. It uses an “Exchange” system we’ve built and it looks like this:  my $exchange = $self-&gt;new_exchange( Preamble( 'deposit' =&gt; { amount =&gt; $amount } ), Steps( Wallet( $self =&gt; remove_credits =&gt; $amount ), BankAccount( $self =&gt; add_credits =&gt; $amount ), ), );  The “Preamble” is a second that declares that messages are going to be displayed to the player and what information, if any, to use for those messages. The “Steps”, however, are only what we’re actually trying to accomplish. In other words, with our exchanges, we mostly write code that describes the desired behavior. All of the “scaffolding” code is hidden away behind the scenes.  For example, note the code we didn’t have to write, but that our exchange system handles:   Exception handling  Transactions  Success messages  Error messages  Logging  Checks for whether or not we had enough money  Taxes (we have them. Long story)   And for other exchanges, we have things such as:   Job queues for asynchronous work  Email  Message popups  ... and many other details I’m omitting from this   In fact, the declarative nature of the above code means that we can even take this “exchange” and cut-n-paste it for our non-technical narrative designers and they understand what it means!  And guess what? We can reuse this. Here’s another example, reusing the BankAccount.add_credits code, for a relatively complex action of buying an item from another player (a bit simplified). But first, imagine how you would write the code for buying a physical item from a vendor and compare that code to the following.  Steps( Location( $self =&gt; is_in_station =&gt; $station ), PlayerMarketItem( $self =&gt; is_not_seller_of =&gt; $item ), PlayerMarketItem( $self =&gt; check_item_is_not_sold =&gt; $item ), BankAccount( $self =&gt; remove_credits =&gt; $amount ), BankAccount( $vendor =&gt; add_credits =&gt; $amount ), Inventory( $self =&gt; add_item_instance =&gt; $item ), PlayerMarketItem( $vendor =&gt; delete =&gt; $item ), )  Did you think of all those steps? How much code would you have had to write to implement all those steps? And would you have remembered all the possible exceptions, the transactions, the SELECT ... FOR UPDATE , and so on? Would you have remembered or cared about all the success or failure messages?  By writing code in such a declarative style, we have taken incredibly complex behavior and not only made it simpler, but more correct.  Here’s another example. In Tau Station, you can “save” your progress by gestating a new clone. If you die, you respawn into your latest close. What does clone gestation look like? It used to look like this mess:  sub gestate ( $self, $character ) { croak( … ) unless $character-&gt;area-&gt;slug eq 'clonevat'; my $price = $self-&gt;price_per_character($character); if ( $character-&gt;wallet &lt; $price ) { $character-&gt;add_message( … ); return; } my $guard = $self-&gt;result_source-&gt;schema-&gt;txn_scope_guard; $character-&gt;dec_wallet($price); $character-&gt;update; my $clone = $self-&gt;find_or_new( character_id =&gt; $character-&gt;character_id, station_area_id =&gt; $character-&gt;station_area-&gt;station_area_id, ); $clone-&gt;$_( $character-&gt;$_ ) for $character-&gt;physical_stats; my $now = DateTime-&gt;now; $clone-&gt;clone_date($now); $clone-&gt;gestation_date( $now-&gt;clone-&gt;add( seconds =&gt; $self-&gt;gestation_delay($character) ) ); $clone-&gt;update_or_insert; $character-&gt;add_message( … ); $guard-&gt;commit; return $clone; }  And that was a bucket of bugs. And hard to follow. Now it looks like this:  Steps( Location( $self =&gt; is_in_area =&gt; 'clonevat' ), Wallet( $self =&gt; pay =&gt; $price ), Clone( $self =&gt; gestate =&gt; $area ), ),  Honestly, which of those would you rather write?  Declarative Exchanges  So how does this magic work?  The Behavior  When you create a new exchange, the first thing it does is go through your steps and figure out what objects might be updated. Then we:   Start a transaction  Refresh all assets via SELECT...FOR UPDATE  Run all the steps  Commit on success and rollback on failure  Notify the player(s) (if appropriate)   It sounds simple, but there’s a lot more going on under the hood. For example, if you are a member of a syndicate and you earn income, you may have to pay “tax” to that syndicate. Thus, the exchange needs to know that it must fetch the syndicate object, lock it, and send taxes to it. As a developer writing the code, you almost never have to pay attention to this. It’s all automatic.  The Structure  Each exchange step looks very similar:  Location( $self =&gt; is_in_area =&gt; 'clonevat' )  In exhange parlance, that’s:  Topic( subject =&gt; verb =&gt; object )  Everything follows a linguistic SVO (subject-verb-object) pattern. To create a new Topic for the exchanges, you create a class called Veure::Economy::Asset:: Topic (there are legacy reasons for the name) and have it inherit from Veure::Economy::Asset . We have another system that automatically finds and loads all these classes and ensures that the asset name is exportable on demand. You just write the code, there’s no need to wire it together because that’s done for you.  Each of these classes takes a subject (the first argument) and implementing a verb is merely a matter of writing a method. The object (in linguisitic terms) becomes the argument(s) to the method. A simple is_in_area check might look like this:  sub is_in_area ( $self, $area_slug ) { my $station_area_id = $self-&gt;station_area-&gt;area_id; if ( $self-&gt;station_area-&gt;area_slug eq $area_slug ) { return $self-&gt;new_outcome( success =&gt; 1 ); } # give them a nice failure message return $self-&gt;new_outcome( success =&gt; 0, message =&gt; ... ); }  Simple, eh? And now we can reuse this to our heart’s content.  Failure  Aside from the fact that the exchanges allow us to write incredibly complex code very quickly, one of my favorite parts is the fact that even though it’s mostly declarative on the surface, underneath it’s objects all the way down. That means we get the full power of OO introspection where we need it. For example, what happens if I’m running the test suite and an exchange throws an exception?  Well, of course, we get a stack trace. And at the top of that trace, we get a stringified version of the exchange. In this case, it’s for refueling a spaceship:  character('ovid')-&gt;new_exchange( slug =&gt; 'refuel-current-ship', success_message =&gt; 'Your ship is now being refueled.', failure_message =&gt; 'Unable to refuel ship.', Steps( Area( character('ovid'), is_in, docks ), Ship( ship('ovid', 'bootlegger'), is_docked_on, tau-station ), Ship( ship('ovid', 'bootlegger'), needs_refueling ), Character( character('ovid'), not_onboard_ship, ship('ovid', 'bootlegger') ), Money( character('ovid'), pay, -15.00 ), Character( character('ovid'), refuel_ship, ship('ovid', 'bootlegger') ), Character( character('ovid'), set_cooldown, {ship =&gt; ship('ovid', 'bootlegger'), cooldown_type =&gt; 'ship_refuel',period =&gt; 1000} ), ) );  In this case, we got an exception because there’s a serious bug: somehow the character has been asked to pay negative credits. This stringified exchange shows this very clearly here:  Money( character('ovid'), pay, -15.00 ),  So it’s dead simple to recreate conditions that cause failures in incredibly complex behavior. In this case, we knew our exchange system was fine, but something was sending it bad data.  Regrets  If there is one regret I have about the exchange system, it’s that it’s not open-source. We absolutely would release this if we could, but when we started on it, it wasn’t clear what the end result would be. Thus, it’s deeply tied to our game Tau Station  . If we find ourselves with the time—or a contract—for this, we will release this for everyone.  As an aside, Ruby has something rather similar, named Trailblazer  . The main difference in exchanges is that we’re not tied to MVC or the web. Trailblazer documents itself as “A New Architecture For Rails”, suggesting a tight coupling. That being said, it has fantastic testimonials which match our internal experience with exchanges. I expect we might see more of this type of code in the future.  ","title":"Making Complex Software Simple","url":"\/articles\/making-complex-software-simple.html"},{"body":" What's a Project Manager?  The Project Management Institute defines project management as follows  :  Project management...is the application of knowledge, skills, tools, and techniques to project activities to meet the project requirements.  As a proponent of agile (in particular, WildAgile ), I find that project management is an oft-ignored area in the field. For example, Scrum defines three roles, the product owner, the team, and the ScrumMaster.   No mention of any sort of management, project or otherwise, is made. It's not that Scrum is against management; it's that Scrum is a framework for getting stuff done and management is an outside force.  So there you are, a brand new project manager (PM), or product owner (PO) and ... wait a minute. What the heck's the difference? In many agile projects, there is no project manager and the PM's responsibility is distributed across the various members of the team: product owner, developers, ScrumMaster (Scrum), Coach (XP), and so on. Thus, the definition of a PM is sometimes muddied and in my experience, varies wildly across organizations.  Curiously, when I find companies have the PO role, that&lsquo;s more clear. The product owner represents the \"voice of the customer\" and maintains the product backlog of tasks that the team works on. The Product Owner, in addition to representing the customer, also represents project stakeholders, so they have two masters, so to speak: the customers and management. But their role is the mama bird role of constantly vomiting up new tasks to feed to the baby birds of the team; they're not really project managers.  But because the waters are often muddied here, I&lsquo;m going to handwave this entire issue and hope you don&lsquo;t notice. Instead, we&lsquo;ll focus on an age-old problem in the PM\/PO space: what should we do next?  So ...  What Should We Do Next?  As a fledgling team lead, PO, PM, or whatever title your company&lsquo;s uses for person responsible for getting stuff done , figuring out what to work on next is a daunting task. It involves having an in-depth knowledge of the project, the product, the business, and the team. And when you're juggling 27 things at once, it can be maddening to have a baby bird cheeping \"what do I do next?\", and demanding a meal.  You might think that backlog grooming\/refinement  is all you need, but that still doesn't tell you, out of the 374 tasks that need to be done, which should be done next . Fortunately, there's a very simple way to do this, and that's to build a business case for each task, using just three numbers.  The Three Number Business Case  A business case is really nothing more than making an argument that a thing should or should not be done, but using numbers to back it up. If you want to learn more, I have a \"rescuing a legacy codebase\" talk that covers this.           The Three Number Problem   But what numbers do we use? For this case, try to pick the three most important numbers that represent value in your tasks.   At one point, when I had the project management role in developing Tau Station  , a free-to-play (F2P) narrative sci-fi adventure, the complexity of the role was overwhelming, but each task's \"three most important numbers\" were clear: complexity, usability, monetization.  The complexity of a task is simply how many \"story points\" (or whatever you call them) that team members have assigned to a task. I recommend using Fibonacci numbers for story point estimates  , especially for what I'm about to show you.  The usability of a task was how useful it was to the customers. Again, you want to use Fibonacci numbers here. You need a feel for what this number is because, in reality, you cannot get a perfect estimate. However, it's clear that a feature which lets customers download reports is more valuable to them than a feature which slightly adjusts the color of a sidebar, so the former might get the number \"13\" while the latter might get the number \"1\".  In the case of Tau Station, everyone is allowed to play the game for free, but as with many F2P games, there are monetization options in the game   and anything which increases the likelihood of monetization gets a higher number. A \"Happy Hour\" feature, where the players get extra rewards when they buy stuff gets a higher monetization number than a feature to make forum posts \"sticky.\" Again, you're using Fibonacci numbers here.  Important: the three things you choose should reflect your business concerns and may very well be vastly different business requirements. You may choose \"security\" or \"legal compliance\" or something else entirely. Tau Station included these as part of the \"usability\" number. Think hard about these numbers because you don't want to get them wrong.  Once you have your three numbers, what next? Well, you need to assign weights to them. This will largely be a process of trial and error until it feels right. In our case, let's say that both monetization and complexity get a weight of 40 and usability gets a 20. Then you play around with formulae until you get a final score, with higher-scoring tasks having a greater priority. Here's the formula I used for Tau Station, though you will need to adjust this for your own needs:  pros = monetization * 40 + usability * 20 cons = complexity * 40 scale = 10 score = ( ( pros – cons ) + 500 ) \/ scale  Let's say a task has a monetization of 1, and a usability of 5, and a complexity of 7. How does that compare with a take of a monetization of 3, a usability of 3, but a complexity of 13? The first task might earn us less money and be less useful, but it's easy to implement. Decisions, decisions ...  Well, monetization and usability are both \"pros\" (benefits), and complexity is a \"con\" (drawback), so we have task 1:  pros = 1 * 40 + 5 * 20 cons = 7 * 40 scale = 10 score = ( ( 140 – 280 ) + 500 ) \/ 10  And task 2:  pros = 3 * 40 + 3 * 40 cons = 13 * 40 scale = 10 score = ( ( 240 – 520 ) + 500 ) \/ 10  Task 1 has a final score of 36, while task 2 has a final score of 22. Thus, we do task 1 first.  Once you learn to plug those three numbers into a spreadsheet, you can then sort the tasks by score and prioritize tasks at the top of the list.    A Spreadsheet Showing Our Three Number Business Case   Real World Usage  When I was doing this, I found it made my work much easier and the team appreciated it because it gave a clear direction. Plus, since the team provides the complexity estimates, they know that they have a real impact on project direction, something that teams usually appreciate.  I used an Excel spreadsheet, but also a small program that I wrote which fetched data from github and ZenHub  to automate the creation of my spreadsheet. Every month I'd repeat the process, fill in missing numbers, sort the rows and share it with the team via a Google Doc. It worked very well and figuring out the project tasks for the month was no longer a stressful affair..  Note that sometimes you will find tasks which don't seem to be scored correctly with this technique. When that happens, investigate whether or not your three numbers seem reasonable. Maybe your weights are wrong, or your formula is off.  Or, as sometimes happens, there are external factors. A task may be complex, offer low usability, and no monetization potential, but another team is demanding that XML feed now and hey, you have to play nice with others. When odd events like that happen, use your judgement instead of the spreadsheet.  At the end of the day, this technique, like all others, isn't perfect. When used correctly, however, it's a great tool for building value and focusing on your real needs. Further, you prioritize greater value and, when asked to justify your decisions, you have numbers to back them up. Even if they're not perfect, quibbling over numbers and formulae instead of hunches is a much safer career move.  ","title":"Project Management in Three Numbers","url":"\/articles\/project-management-in-three-numbers.html"}]
\ No newline at end of file
+[{"body":" I have extensive experience with:   Agile project management.  Hiring and managing remote teams.  Catalyst, Moose, Template Toolkit, DBIx::Class MVC stacks.  Testing of large-scale applications.  Fixing database integration issues.   I work in many areas, but if it’s Perl-related and your company uses Perl heavily, there’s a good chance that your employees already know me by reputation or have read one of my books. You’ll find me easy-going and diligent. I have a strong background in software testing, having built or worked on many of the most popular testing tools for Perl, including my favorite, Test::Class::Moose , an xUnit framework for large application test suites.  Email me and let’s discuss how I can help you build value for your customers.  A few things I have been hired for:   Building software development teams.  Corporate event keynotes .  Seminars to help companies transition to agile ( sneak preview ).  Rescuing legacy codebases .  Building new software systems in Perl .  Rebuilding or fixing test suites .  In-house A\/B testing (proprietary).  Pharmaceutical ETL systems (proprietary).  Real-time bidding (proprietary).   You might also want to check out:   My LinkedIn profile  My Github account  My CPAN account   Here are some of my articles to help you get a good sense of whether I’m a fit for your needs.  ","title":"Hire Me","url":"\/hireme.html"},{"body":" I’m Curtis “Ovid” Poe. I’ve been building software for decades. Over the past few years, I’ve moved more to the business side, managing projects and teams, training companies how (and when) to be agile, and still writing software and books about software development. I regularly speak at conferences and corporate events across Europe and the US.  I specialize in building database-driven websites and teach developers how to design and use databases. I also specialize heavily in the Perl programming language and joined The Perl Foundation board of directors in 2009.  Useful Links   LinkedIn profile  Github  Twitter  Facebook  All Around The World   Hire Me  Drop me a line and let’s discuss how I can help your company. Naturally, I do a lot of bespoke development on large, database-driven websites using Perl, but there are many other areas I can help you with. I can help get your legacy codebase back on its feet or fix your database issues . I also do agile consulting and can help you understand when to choose Agile . In fact, I’ve been working in Agile (mostly Scrum and XP) for so long and consulted with so many company that I’ve created “WildAgile”, an agile system that maps to how your developers actually work , rather than imposing an artificial, monolithic structure.  Public Speaking  I also speak at private corporate events and technical conferences across Europe and the US and am frequently the keynote speaker. Here’s a video of one of my talks at a Perl conference, but was also rated 5 out of 5 stars at OSCON :     ","title":"Ovid's Programming, Politics, and Prose","url":"\/index.html"},{"body":"","title":"Public Speaking by Ovid","url":"\/publicspeaking.html"},{"body":"","title":"3D Starmap in JavaScript","url":"\/starmap.html"},{"body":"   Tau Station’s universe is dangerous   You’re doing heater runs in Taungoo when a contact you haven’t heard from in cycles asks you to quietly deliver a package to Nouveau Limoges, another station in the Sol System. The money’s good so you head down to the port, hop in Amethyst, and launch. Amethyst’s an older ship and she’s higher maintenance than you’d like, but she keeps flying.  A little over 7 segments later (just over 1.5 hours, old Earth time) you arrive at Nouveau Limoges and that’s when the trouble kicks in. You’re a Consortium citizen, but Nouveau Limoges is a Gaule station and you forgot to renew your visa. Immigration computers notice your status and Amethyst’s nav system is overridden to auto-deport you. Meanwhile, you were supposed to deliver the package in your hold within 8 segments and now it’s starting to change shape. You think you’ve been set up.  Years ago I came up with the idea for Tau Station  , a free-to-play sci-fi narrative adventure you can play in your browser. The intent was to create a universe you can live in, online. As of this writing, we’re still in alpha, but it’s an open alpha and I encourage you to check it out. We’ve developed some fascinating technologies for it and I’ve given a few talks about it. Here&lsquo;s one I gave at FOSDEM  , one of the world’s largest open source software conferences, held annually in Brussels, Belgium, but I like the following one in Amsterdam better:     The Catastrophe  About two centuries ago (Galactic Coordinated Time), civilization was destroyed. No one knows why, though there are many theories.  After breaking free from the confines of Earth, humankind spread throughout the galaxy, setting up colonies on distant planets and hollowing out enormous asteroids for use as space stations. A golden era of post-scarcity was attained. However, humankind was viciously attacked en masse in an event known as The Catastrophe. Planetary defense systems were turned on the planets they were defending. Stations across the galaxy had their air vented, life support systems shut off, and databanks purged. Reactors went critical on ships. Humanity, across the galaxy, was driven to near-extinction in the space of just a few segments. Then the attack abruptly stopped, for reasons unknown.  The Aftermath  In the aftermath, planets were unreachable by those remaining few who were sprinkled out among the stars. For every station with survivors, there were at least a hundred more without. Most human knowledge was destroyed. Books were an anachronism, so the databank purge wiped out most of what humanity needed to continue surviving, as well as the records we had of our past. Those survivors with skills were quickly called upon to spread their knowledge, but their expertise was piecemeal. Without the massive databanks, human technological advances came to a halt. There have even been disputes about the current year, but it’s believed to be the middle 2600s.  Today, no one knows the entirety of how to build a starship, or how to construct one of the massive wormhole jump gates that many star systems have. However, we’ve retained or relearned enough to at least repair them — if we can scavenge parts from dead stations or ships.  This is humanity today. Most of our history has been lost. Much of our existence is a hardscrabble lot, and we few left trying to recover and to simply stay alive. Humankind’s reign has been reduced to a 40 light-year sphere, centered on Sol System. Contact with the planets and most of the other star systems has been cut off. Humans are slowly rebuilding their stations, in hopes that one day, we can regain our former glory.  But who attacked humanity and why? More importantly, are they coming back? There are many theories, but no proof or any solid evidence. Whatever the cause, most of humanity have been raised to be on permanent alert. We’re rebuilding, and if they attackers return, we want to be ready.  The Game  Tau Station is a narrative game. That means it’s a game you read. There are no adolescent boys camping spawn points and screaming improbable things about your mother; It’s a game that lets you take your time, think, and plan.   And because it’s also an MMO, you can interact with other citizens of the Tau Station universe, join a syndicate, explore new star systems, and watch with us as we build the universe.  However, just because it’s text doesn’t mean it’s ugly:    From our tutorial mission   Tau Station is ultimately a murder mystery, with most of the human race being the victim. As the game unfolds, we’ll be revealing more and more of the story, letting players discover for themselves who was really behind The Catastrophe. It’s also science-based (though not hard science fiction). There is no artifical gravity or FTL travel. We use E = mc 2 and the mass of your ships to calculate the amount of anti-matter necessary for travel. We use proper orbital mechanics to calculate the orbits of space stations. And while we have to take shortcuts here and there (it’s a game, after all), we stay faithful where we can.    We still have a lot to do, including:   Syndicate Buildings  “Elite” style trading  Asteroid and ice mining  Ship expansions  Hundreds of new stations  ... and much, much more.   Come join us and see for yourself!  We’re still in alpha, so you can expect changes to come, but we’re pushing hard to create the features we need for launch. I hope you enjoy it; it’s really a labor of love for all of us.  If you want to learn more, read our blog  and checkout out \/r\/taustation on Reddit  .  ","title":"Tau Station","url":"\/tau-station.html"},{"body":"   Preface  The Problem  Background  What is WildAgile?  Benefits of WildAgile  The Workflow  Assumptions Made in Creating WildAgile  The Details  WildAgile Roles  WildAgile Process  WildAgile Remote Teams  Discussion  Extending WildAgile  WildAgile-Compliant  Glossary  Summary     Preface  Some of you may be familiar with Alistair Cockburn  , one of the founders of the modern agile movement. I recently had a brief email discussion with him regarding WildAgile. After reading this article, he replied (and I’m quoting with permission):  I think you should continue with what you’re doing, and get more companies to go along, that will be really great for the world.  WildAgile is an attempt to return agile to its agile roots. So let’s get started.  The Problem  I’ve worked with a variety of successful companies, many of which are Agile in nature, but when discussion of Agile arises, the conversation often goes like this:  Them: We’re agile.  Me: Ah! Do you use Scrum, XP, something else?  Them (embarrassed): Um, er, we just sort of get things done.  And they do! And yet they’re embarrassed by the lack of a name for what they actually do. I often hear something like “we do a modified scrum”, meaning they have sprints and stand ups and that’s about it.  Thus, I developed WildAgile , an Agile methodology that is so agile that it describes what actually happens in the wild instead of what you are supposed to do. It’s designed for very rapid delivery, with Just Enough Process to provide order to the chaos. Though I draw inspiration from Scrum, XP, Kanban and Crystal Clear, there’s a good chance that claiming your team is WildAgile-compliant merely involves reading this and saying “yup, we’re WildAgile-compliant.”  All bolded terms are explained in the description and also in a short glossary near the end. There aren’t many bolded terms because I focus on process, not terminology. The “WildAgile Roles” and “WildAgile Process” sections are all you really need to read to understand WildAgile. Everything else is just providing the rationale. This documentation of WildAgile is released under the CC by 2.5 license . This means that you’re free to do anything you want with this documentation, including sell it, so long as you credit the original authors (just linking to this page is fine). See the license description for details.  Background  When teams transition into an Agile environment, the changes can be intimidating. XP practitioners regularly complain about pair programming (and hence don’t do it). Scrum practitioners often ignore retrospectives. System administrators get upset about a “failed sprint” because they had to stop all work to fix a critical security issue.  And yet work still gets done. Developers finish a task and take a new task. Wash, rinse, repeat. By formalizing this I introduce Just Enough Process to get things done in a structured manner. “Just Enough Process” is an appropriate tagline for management. “Shut Up And Write Some Code” is an appropriate tagline for developers.  What is WildAgile?  WildAgile is appropriate for small teams where the entire code base can be roughly understood by a single programmer. That doesn’t mean that someone points to a given library and the developer knows how it’s coded, or exactly what it does, but the developer should at least have a sense of what it’s for. Because WildAgile is designed to avoid bus-sensitive code , this might be better for small to medium sized codebases, though I know of a huge company, with a codebase exceeding one million lines, which is very profitable and follows a process not quite unlike WildAgile, despite having hundreds of developers. What this means is that as a developer transitions from Apprentice to Veteran (described later), they won’t always know how to solve a problem, but they’ll know where to look to solve that problem.  You may think of WildAgile as a transitional methodology. This means that you may adopt WildAgile before your team transitions to a more formal method (it’s particularly well-suited as a transition to Scrum), but many teams find that WildAgile is all they need.  Benefits of WildAgile   High morale: developers can focus on getting stuff done.  Rapid development: developers can focus on getting stuff done.  Ease of implementation: developers can focus on getting stuff done.   You may notice a theme in this.  That being said, WildAgile is not without its risks. These will be discussed on an ad hoc basis as I discuss its components. However, I do not recommend WildAgile when a potential for real harm occurs. For example, I probably would not use WildAgile to design guidance systems for planes or medical radiology software. I also do not recommend WildAgile for highly complex projects in which the majority of developers cannot understand how various components fit together.  The Workflow  Just to make it clear what I’m focusing on, here’s the development workflow for one company I’ve worked with:    Everyone ignored this and still got work done.   Every task is expected to follow that process. If you have to break a task into new tasks, you’re still expected to follow that process for every subtask. Developers in this company regularly ignore this workflow and still get things done, but management gets frustrated because they can’t see the exact step that every task is at.    WildAgile strives to implement the following workflow:   Things are not this simple in the real world, but I strongly advise clients to understand the Platonic Ideal of Agile and constantly strive towards it.  Assumptions Made in Creating WildAgile  Drawing on work from Alistair Cockburn’s Crystal Clear methodology, I assume that the following three requirements are the most important in developing high-quality products.   Easy and regular team communication  Rapid delivery  Expert customer access   Many experienced individuals feel that if these three requirements are met, a positive-feedback loop is created which allows the team to rapidly build and improve upon products. Any obstacles to these three requirements being met are detrimental to the productivity of any WildAgile-compliant team.  I am also a firm believer in the Pareto Rule: 80% of your benefits will stem from 20% of your actions. WildAgile attempts to implement that 20% of actions that give you majority of benefits of Agile. In fact, I’ve found that for many smaller teams, attempting to implement the other 80% of actions may actually be detrimental as you spend more time fighting process instead of developing software. This is due to a fundamental tension in many Agile methodologies: it’s an attempt to create processes that are designed to minimize process. It’s a delicate balance that is easy to get wrong.  Cockburn’s (pronounced coe-burn , if you’re curious) team requirement actually said “colocation” of the team, requiring team members to be physically near each other. This ignores today’s reality. Savvy companies realize they have access to an entire world of developers if they are willing to have remote workers. Thus, tools such as Slack, Google Hangouts, Skype, Zoom, and other communication channels should be employed to ensure that someone is always available to help. I’ve seen WildAgile work in teams that are also temporally diverse (separated by many time zones), but this does introduce challenges that require extra attention to team communication.  Note: if you don’t have much experience with remote teams, please read my Managing a Remote Team article.  The Details  Yeah, yeah. That’s enough blathering. What’s it all about?  WildAgile Roles  There are only three roles defined in WildAgile.   Developer  Team Lead  Product Owner   Developers , of course, develop the product. They take tasks , finish them, and then take new tasks.  The Team Lead (often shortened to “Lead”) is also a developer. The Lead is also the expert who knows the entire system and can be relied upon to answer questions and provide excellent feedback. Developers decide how they’re going to implement something, but if there is a problem, the Lead has final say. The Lead should defer to the Developer whenever possible and offer constructive advice when appropriate. Team Leads support, not direct.  The Product Owner ( PO ), similar to Scrum, maintains the product backlog . However, there is no “sprint backlog” because there are no sprints. The PO is responsible for knowing the full backlog and being able to answer any business question related to them. It’s preferred if the PO can also answer technical questions as well, but I realize that it’s often hard to find someone who is both a business expert and a technical expert. Thus, the PO focuses on business excellence and the Team Lead focuses on technical excellence.  Note: if you’re new at this and you’re overwhelmed by prioritizing the product backlog, read my article “Project Management in Three Numbers” . It makes this much easier.  Additionally, the PO serves as the proxy for the customer. They are the “expert customer” that you should be able to rely on at all times to answer questions about the product. The PO is encouraged to practice an embedded anthropology approach to knowing their customers. This means that they should use the product, but also work directly with real customers (both new and existing) to understand customer needs and desires.  As an aside, the customer should always be referred to as “the customer” and never as “the user.” In English, the word “user” can have a negative connotation and in our experience it is often used in a condescending manner. Customers are people too.  WildAgile Process  Every day, a 15-minute time-boxed stand up is held. It’s recommended to be at the end of the day to ensure that what you did is fresh in your mind, but this is not a requirement. The stand up begins with the Summary . Before the stand up, the product owner should briefly discuss where you’ve been and where you are going. Any interesting future issues for the team should be mentioned at this time. The benefit of the Summary is that it helps remind developers of the context of their work and better understand upcoming priorities. This makes it easier to choose appropriate tasks.  After the Summary, each team member should explain:   What they did today  What they’ll do tomorrow  Any blockers they have   On a daily basis, every team member is allowed, even encouraged, to improve the code base. Add a Makefile target to rebuild the project. Factor out common code. Move test data into fixtures. The developer is expected to mention this in the stand up and it is the responsibility of the Team Lead and the PO to rein in developers who spend too much time “improving” instead of “developing” (and vice versa!).  Tasks are “things a developer can do.” Every task is done in a separate branch (when appropriate) and when it is done and the tests pass, it is merged back into the “master”, “trunk”, or “integration” branch, or whatever you call it. Tasks are expected to include tests, when feasible, and if merging the code breaks any test, the developer responsible for merging the code should be the person responsible for fixing the tests (again, when feasible). Details about how to manage source control are left to the team.  Tasks should be as short as possible. Any task expected to take over a certain amount of time (a week is a good limit) should, if possible, be broken down into smaller tasks, each of which can add value even if the overall task is cancelled.  When a task is finished, the developer independently chooses their next task, keeping in mind the context provided by the last Summary. They should prioritize tasks assigned to them, high value tasks, and quick tasks. It is the responsibility of the Team Lead and PO to guide developers to suitable tasks if the team member is regularly making poor choices on their next task.  If enough open tasks are open, it is the optional responsibility of the PO to designate Priority tasks. If a new task is to be chosen, they should be chosen from Priority tasks. If a team member is unconvinced of their ability to accomplish any Priority tasks, they should speak to the Team Lead for guidance.  Obviously, if a critical issue arises, any task can be stopped and the Team Lead can assign someone to tackle the issue. There is no formal structure to this because WildAgile is designed to be as lightweight as possible.  New team members ( Apprentices ) are initially encouraged to focus on a single area to become very comfortable with the business rules. After the Team Lead and PO agree that the developer has become comfortable in their area, they become Veterans . Veterans are strongly encouraged to regularly choose tasks in areas of the code base they have less experience with. If Veterans do not regularly work on different areas of the code, they may find themselves focusing on a small subset of the code base. Not only does this limit the Veteran’s ability to contribute to the code base, it encourages other developers to not work on areas of the code that it’s assumed the Veteran will work on. It is the responsibility of the Team Lead and the PO to ensure that Veterans are regularly choosing tasks outside of their area of expertise in order to spread knowledge of the system.  Team Leads and POs do not promote Apprentices to Veterans until the Apprentice:   Appears to understand the business (not the code base!)  Is writing solid code  Is writing appropriate tests   Releases are not described in the WildAgile process, but it’s suitable for both continuous deployment and regular releases (as one might expect from a sprint or iteration).  Note that in our experience, Apprentices get promoted to Veterans fairly quickly and in established teams, you generally only have Veterans, thus reducing the distinction between the two.  WildAgile Remote Teams  Many companies are discovering the competitive advantage of having remote workers:   You have access to a much wider pool of developers.  You don’t need to pay for office space.  Developer salaries are easier to control.   That last point is what is driving much of this and is causing some industry disruption. When you realize that your $150K a year developer can be had for $30K, that opens up possibilities you may not have had before, particularly when you find that expert you couldn’t find locally.  However, companies often feel uncomfortable with remote workers because they’re less used to this. WildAgile works well with remote workers, but a key change needs to be made: stand ups.  For colocated teams, time-boxing a stand up to 15 minutes works well because if there are issues later, a developer can simply walk over to someone and ask. That doesn’t work with remote workers. Instead, you simply extend the stand up time. I recommend 30 to 45 minutes. Yes, this means you’ll be spending more time in the stand up, but you can communicate more about what you’re doing, and how you are doing it. This helps to catch problems early on and minimizes the need for developers to distract other developers while they’re working.  Discussion  The idea behind WildAgile is not to formalize a new methodology, but to put a name on a practice that is already moderately well-known but followed in an almost embarrassed manner. It’s a way of following a set of reasonable development guidelines that can eventually move on to a more appropriate Agile development process. Think of it as “Agile light”, if you will. By putting a name to it, you can say “we are WildAgile-compliant” instead of “eh, we just get things done.”  In short, WildAgile eliminates the silo problem of cowboy coding, but allows the extremely rapid development of products. It encourages creativity while having Just Enough Process to keep development on target.  By separating developers into Apprentices and Veterans, I create the psychological incentive to explore more of the code base. Ambitious developers quickly want to shed the Apprentice label and fully participate in the rest of the team’s work. However, by starting out by becoming an expert in one part of the system, the Apprentice can quickly gain confidence in their ability to participate and make a meaningful impact on the system.  Hiring for a WildAgile process is important. You want developers who are comfortable in a fluid environment, but can also appreciate business concerns. There is a tremendous amount of freedom in WildAgile and it’s very easy for someone to get carried away and try to “perfect” (*cough*) everything at the expense of adding value to the company. Thus, both the Team Lead and the PO are required to keep things in check.  Also, WildAgile benefits from skilled, conscientious developers. It’s very easy to spend too much time “perfecting” your code, but it’s also easy to spend too much time pushing out new features and ignoring technical debt. This is a hard balance to maintain. WildAgile does not address this directly because every project is different and every company is different. However, as a guideline WildAgile suggests that the PO advocate for features and the Team Lead advocate for technical debt management, but each should be able to understand and even agree with the other’s point of view. If you can’t balance these two needs, you’re likely to have long-term problems.  In practice, I find this development model to be very successful for small projects. The pace of development is very rapid and I don’t find that code quality is significantly worse than for other methodologies. The most significant obstacle that I find is appropriately managing the backlog. Typically backlog items are merely added to a ticketing system, such as Jira, RT, Redmine or Trac. This does not scale well unless the PO is competent at managing the tickets and creating appropriate Priority tasks.  Extending WildAgile  Do you like retrospectives? Do them. Do you need milestones? Great! Do you like pair programming? That’s fine. WildAgile is designed to describe the core of what successful development teams are already doing. Items outside of this core are deliberately unspecified to allow you to customize WildAgile to your heart’s content. A guiding principle of WildAgile should be “whatever works for you.”  You are encouraged, nay, required to customize WildAgile to meet the needs of your team.  WildAgile-Compliant  Many teams are already WildAgile-compliant without even knowing it. If you follow Scrum, you’re probably WildAgile-compliant. If you follow XP, you’re probably WildAgile-compliant. If you do ad hoc development, you may already be WildAgile-compliant. Don’t stress too much about it. I don’t envision WildAgile-compliance courses. I don’t envision WildAgile-certifications (though if you pay me enough money, I’m happy to give a one-hour presentation and print up WildAgile-certifications for you).  Glossary   Apprentice A new developer becoming an expert on part of the system.    Backlog Tasks which have not yet been done.  Critical Issue Something which must be done now .  Just Enough Process Only enough process to provide structure.  Priority Tasks which should be taken before other tasks.  Product Owner (PO) The proxy for the customer. Maintains backlog and assigns Priorities.    Stand up An end-of-day “what I did, will do, and blockers” explanation for every developer.  Summary The PO’s business-focused summary of the project status, given immediately prior to the stand up.  Task Something a developer can do.  Team Lead The code base expert. Should also understand business concerns.  Team All developers, including the Team Lead.  Veteran A developer who is allowed to work on all parts of the system.  WildAgile “Real World” Agile.   There is very little terminology and most of this fits what you already know.  Summary  Shut up and write some code.  Seriously, there is nothing radical about the above and it mirrors what people are actually doing rather than forcing them to adopt a new methodology all at once. You might view it as a gentle transition to more formal Agile methodologies. Or, if it’s working fine for you, it might remain your Agile methodology. Its very loose structure presents some risks, but a key component of managing risk is being aware of it.  ","title":"WildAgile","url":"\/wildagile.html"},{"body":"    Why am I Writing This?  1984  Cheating on Tests  Cheating on Homework  The Break-In  Interlude: The Tax Office  Getting Caught  Aftermath  Tax Office?     Why am I Writing This?  This is here because when I reread the eulogy for my father , I realize that no one knew much about his life. This isn’t here so much as to share my life with the world, but to start sharing snippets of it with family members after I pass. There might be some bad things here, but I know how painful it was to try and reconstruct my father’s life, so I don’t want my family to go through the same thing.  It might prove to be mistake. I had a rough life growing up and some stories I might share do not reflect well on me.  1984  This is the story of how I got caught breaking and entering in my high school in Texas. Names of guilty parties have been changed because most of them are still alive and it’s not fair to taint them with high school stupidity. Also, this happened 30+ years ago, so it’s possible my memory is shaky on some timelines.   It will come as no surprise to most readers that teenagers are stupid. There’s interesting research explaining why  , but it boils down to having brains that aren’t fully developed  until their mid-20s. I think mine didn’t develop to my mid-30s, but that’s a story for another day.  I had been referred to in high school as a “goody two-shoes.” That’s because, well, I had a reputation for being what we in France refer to as « carré ». The word translates to “square”, but in this context, it tends to mean “someone who is correct and doesn’t break the rules.” (It’s possibly less pejorative than “straight-laced.“)  In short, I was a rather unpopular nerd. I didn’t smoke, do drugs, swear, and when I was once asked if I masturbated, I instantly replied “of course not!”    I was as pure as the driven snow, except for my militant atheism thing. But otherwise, I simply didn’t break the rules.  So it was rather a shock when students found out that I, too, was a rebel.    Yeah, that’s me looking like an idiot on the upper left.   Cheating on Tests  My first high school, Keystone  , was a private high school for the academically gifted. My grandmother, who had been paying for the school, pulled me out after two years because, frankly, I didn’t care about my grades. I didn’t have a great childhood, so I put only enough effort into school to not fail. But my geometry teacher took pity on me and let me play around in the computer room, a room reserved for grades 11 and 12.  I taught myself BASIC and when I was returned to Marion High School  in Marion, Texas, I found myself in a much easier school and suddenly, I was getting A’s. It helped that I was hanging out in the computer room and was able to get copies of the teacher’s tests. Another student we’ll call “Alex” was a teacher’s aid and he’d swap the tests he would type up for my class with the tests I would type up for his class.  But one day, he said “no” to the mid-term. Seems the teachers were getting wise and put pressure on him, even though he didn’t admit it. But he also told me that it didn’t matter because his disk had crashed (in the 80s, we’d use floppy disks that easily got corrupted) and he couldn’t print out our mid-term even if he wanted to.  No matter. We couldn’t keep our floppy disks, so we had to return them to the desk when we left the room. I had already acquired the master (root) password for our TRS-80 computers and I was able to retrieve his disk and do a raw disk dump straight to the dot-matrix printer. I had most of the test, though some of it was corrupted (the essay questions).    An old, TRS-80 computer with dual floppy drives. Source    I gathered several students and we hit a truck stop restaurant to study for the mid-term. And then one of them said, “I really like Mrs. Womack. I can’t cheat on her test.” And one by one, all of the students agreed we couldn’t cheat. I did, too, but it was because I was lazy.  So we had a nice time at the truck stop and the next day, I didn’t get an A on my test.  I later found myself sitting at a computer (surprise!) after physics class when Mrs. Womack burst in, furious.  “I thought I could trust you, Curt!” (Everyone called me “Curt” instead of “Curtis” back then). “I just found out you had a copy of the mid-term.”  I looked at her and calmly replied, “if I had a copy of the mid-term, would I destroy my GPA like that?”  Her eyes widened and she stopped. And then she apologized. Of course, it was stupid that I would do such a thing. I never heard another word about it.  Cheating on Homework  Of course, then, as now, I loved computers. One of our assignments was doing ten key by touch  . To say that it was boring would be an understatement. We’d start with seeing a random series of ones and zeroes and have to type them in as quickly as possible. When we had a good enough score and speed, we’d move to zeros, ones, and twos. And so on.  I wasn’t going to do this. Since I had the master password, I could read and analyze my result output. After doing that, I wrote a program that would simulate my error rate and typing speed and simply write out my results so I could go on to playing with the computer instead of doing boring work .  Today, of course, I’m horrified by that. I should have analyzed the standard distribution of my typing speed and error rates and ... I mean I would never have done such a dastardly thing. But hey, I was a teenager.  So while the other students in the class were slogging through their ten-key exercises, I would run my covert program and go back to playing.  But ... these were TRS-80 computers with dual floppy drives. You put the master floppy in the top and your floppy in the bottom. If you got it wrong, your scores wouldn’t save.  Mine would because I had the master password. And if I had the disks reversed, I would overwrite the master floppy. And I had gotten the disks reversed. “Oh no!”  Mrs. Bode (pronounce boh-dee) looked up and asked, “what did you do, Curt?”  I had played around enough that I had “crashed” quite a few floppies and was told that I would not be allowed back in the computer room except for proper classes if I did it again. My first thought, after realizing I would be caught cheating, was that I wasn’t going to be allowed back into the computer room. Disaster!  “Nothing, Mrs. Bode. I crashed one of my own disks, not yours.”  Naturally, I had overwritten the master disk and while I could cover up my crime, I couldn’t hide the fact that I had crashed another school disk. The Master Disk. I was going to be barred from the only thing in school that I truly loved.  So I approached “Alex,” my erstwhile partner in cheating and explained the problem ... and my plan. He listened and said he wouldn’t help.  I was stuck. So I approached another friend, “Bob.” Bob was a player in the AD&amp;D (Advanced Dungeons and Dragons) campaign that I was running and he was instantly eager to help with an adventure!  Except we needed transportation. Enter “Charlie.” Charlie was also a player and agreed to help, but with a caveat. For (identifying) reasons I won’t discuss, he was much more at risk if he got caught, so Bob and I agreed to keep him out of it.  The Break-In  That night, Charlie picked me up at my house and we went and got Bob. Charlie was driving a pickup truck with a ladder in the back and as we got to the school, Bob and I took the ladder and Charlie drove off. He agreed to come back in an hour.  Bob took the ladder and placed it on the side of the school and he and I climbed to the roof. At the time (I don’t know about today), there was a large, open-air space in the school—see the photo above—and if you dropped down into that from the roof, you were inside the school.  The locks were shabby and anyone with a pocketknife, something most farm boys carried at that time, could jimmy a lock with ease. So Bob shimmied down a column into the school and immediately gestured for me to stay where I was. Seems that in the building next to us, but clearly visible down the hallway, was the band practicing. Bob grabbed a desk that had been left lying around, flipped it on its side and crouched behind it while he worked on the lock.  When he got it open, he motioned for me to come down. I shimmied down a column and when he motioned the coast was clear, I crept into the classroom after him.  Of course, we were wearing gloves as we rounded up every floppy disk we could find. But I knew that Mrs. Bode might remember the incident with me, and I was paranoid. So Bob wrote on the board “Mrs. Bode is a bitch.” Bob, like Charlie and myself, was such a “good” students that there was no way anyone would suspect us of writing that. With no witnesses, no fingerprints, and a message to throw them off the scent, we committed the perfect crime.  As we left, we hid in the bushes with the disks until Charlie showed up again. We clambered into the truck and drove off. Bob kept saying over and over again about how exciting that was, just like a D&amp;D adventure. Charlie and I shared a look. We knew then that we were screwed.  “Bob, you can’t tell anyone, right?”  “No, I won’t tell anyone.”  Interlude: The Tax Office  Let’s take a little diversion. At this time, I had heard plenty of rumors that our school had been doing dodgy things to get more tax income. Schools in the US are generally funded through income taxes, sales taxes, and property taxes. If you live in a poor area of the US, your school is miserable because it’s underfunded.  Rumor had it that our school had figured out a way to get more property taxes. In fact, our school even had its own “tax office.” It was a room in the school that was filled with computers, that students didn’t enter, and allegedly helped with collecting taxes for the entire Marion School District. How much of this is true, I have no idea, but hey, rumors! They’re chum for the adolescent sharks.  The tax office door, unlike the other doors in the school, had a steel plate blocking off the latch, so you couldn’t easily jimmy it open with a knife. Not a problem. I had a acquired a key. And I knew where they had written down their computer passwords. I was going to break in and bust the whole corrupt scheme wide open. Like Bob, I was hungry for adventure.  I never did break into it, but it became relevant later, by a very weird coincidence. There were a couple of weird coincidences that day and the reason we got off so lightly may have been because the school had other matters to worry about in what had to be a complete day of hell for the school.  Getting Caught  The next day at school, everyone was talking about the break-in and it was scary. Bob, Charlie, and I gathered around and I reiterated my position. There were no witnesses, we had worn gloves, and no one had any reason to suspect us. No one is to say a word to anyone, right? Bob and Charlie both agreed.  At the beginning of physics class, Bob was called to the office. Charlie and I looked at each other and knew we were screwed.  Later, Bob came back. He was in tears.  A word about Bob. Bob was a great guy. He was a jock, but a tall, good-looking, extremely intelligent jock and an all-around nice guy. He didn’t deserve this, but I had gotten him into it. Bob had previously caught by his father of committing the heinous crime of playing AD&amp;D with me and this was going to be the last straw. Bob’s father was going to mad and Bob was terrified.  A quick, in-class whispered consultation revealed what happened. Bob had told “Dan” who promised he wouldn’t tell anyone. Dan had told “Evan” and, as near as we can figure out, Evan told everyone.  Had Bob given up Charlie? No, he kept his name out of it. But yeah, the school knew about me. Good, I wouldn’t mention Charlie either. It was my fault and I didn’t want him to get in trouble too. He had too much at stake (I’m deliberately not saying why because it’s identifying).  So when physics class ended, I was rather curious as to why I wasn’t called to the office. Yeah, they knew what I had done, but they hadn’t called me down to the office. What the hell? Something odd was going on. So I skipped my next class and went to the office. When the secretary saw me I simply said, “I think the principal wants to see me?”  She said, “yes,” and pointed to his door.  The principal wasn’t there, but our history teacher was there. He was writing a letter that he quickly covered up before hastening out of the room. Seems it was his resignation letter. He was caught in a compromising situation that, at that time, wasn’t a crime, but was serious enough to demand his resignation (not, I’m not going to say what happened). The principal was, of course, a touch distracted by this.  The principal arrived, along with the school district superintendent. They sat down and began to describe what they knew (a tactical mistake, because it told me how to play my story). Two students had broken into the school, the police were called, and there were witnesses and fingerprints. What did I want to say before the police came?  Well, that was interesting. Two students, not three, so Bob had kept his word. This was in the middle of nowhere so, unless we were spotted by someone at band practice—something I felt was unlikely—they lied about witnesses. And I knew they didn’t have fingerprints.  So I told them the truth, the whole truth, and nothing but the truth. Except that my truth left out Charlie. Out of this entire mess, Charlie was never caught.  “You’re lying.”  “No, I’m not!”  “You broke into the tax office!”  The hell? At this point, my mind is reeling. Yes, I had planned to break into the tax office, but I hadn’t done it yet. Had someone else broken in the same night? Had the school found out about my plan to try to expose tax fraud? Had the person who gave me the key told them how I got the key? What the hell?  “I didn’t break into the tax office!”  “We have fingerprints. You’re going to be arrested.”  I continued to protest my innocence and they continued to insist that I did it until I got mad and yelled, “I didn’t do anything!”  “You stole the floppy disks.”  “Oh, yeah.”  Aftermath  Charlie was never implicated. Bob and I received in-school suspension, were barred from the class trip, and I apologized to Mrs. Bode, explaining that we only wrote that she was a bitch because we knew no one would believe we thought that. She was surprisingly gracious about it. Apparently, she understood teenagers better than teenagers do.  And as our final punishment, Bob and I were forced to complete the ten-key by touch homework for every other student in the class. All of our free time was spent in the computer room doing that. I got rather good at ten-key by touch and can still do it to this day.  Of course, I worked out the math. I pointed out to Mrs. Bode that it was literally impossible for us to finish all the students ten-key assignment by the end of the year. But she was unmoved. We had to do it.  And then she was gone for a week. We had a substitute teacher and I, not having properly learned my lesson, took the program that I wrote and hastily completed most student’s work by the end of the week. When Mrs. Bode returned, Bob and I only had a few more day’s worth of work to do.  I’m pretty sure that Mrs. Bode knew that we had cheated, again. But she never said a word.  Tax Office?  So that’s the end of the story, but you might be wondering what the hell was going on with the tax office break in.  As I found out later, there wasn’t one. That’s what the school wanted to have me arrested for. As it turns out, the school’s computers couldn’t connect to their tax system and the school thought I had something to do with that. I had been planning it, but they didn’t know that. Instead, when I was breaking into the school, AT&amp;T was breaking up  . Due to an antitrust action, AT&amp;T was forced to break up into seven smaller phone companies and did so when I was breaking into the school . The next day, the school’s computers appeared to be “broken” and given that I was the school “computer kid” who kept messing with computers, it was assumed that I was responsible.  ","title":"How I Got Caught in High School","url":"\/blog\/how-i-got-caught-in-high-school.html"},{"body":"    Forward  Accelerationism  Techno-lust  Free Markets  Capitalism  Market Discipline  Luddites     Forward  In the vein of Ted Kazinsky, Karl Marx, and others before him, Marc Andreessen has proudly penned a manifesto, The Techno-Optimist Manifesto  . Or more properly, “a rant.”  I chose the word “rant” with care. He’s ranting against something, but it’s a terrible essay riddled with citations from racist writers (whom he admires), straw men, false dilemmas, and reads like a high school sophomore barely squeaked out a passing grade in a macroeconomics course.  Were it not for the fact that it’s authored by a famous billionaire, it would just be another screed on another blog no one reads (like this one).  Accelerationism  The central thesis is the call for “accelerationism”:   We believe in accelerationism – the conscious and deliberate propulsion of technological development – to ensure the fulfillment of the Law of Accelerating Returns. To ensure the techno-capital upward spiral continues forever.   It’s possible that Andreessen isn’t aware that fascists and white supremacists love accelerationism  , but given all of the authors he cites, including admiration for “the father of accelerationsim,” Nick Land, I strongly doubt this. Nick Land, in particular, is often accused of advocating for “scientific racism” and eugenics  .  Andreessen also calls for free markets and for people to stop hating technology. The following assumes that he’s arguing in good faith and not deliberately using deceptive arguments. Also, libertarians are, demographically, overwhelmingly white males from rich countries and have long harbored white supremacists  , but there’s too much to unpack there, so that is (unfortunately) being left our of this response.  Techno-lust  Let’s get the easy one out of the way.  Andreessen uses the word “technology” 48 times in his rant. He opens with this:   We are told that technology takes our jobs, reduces our wages, increases inequality, threatens our health, ruins the environment, degrades our society, corrupts our children, impairs our humanity, threatens our future, and is ever on the verge of ruining everything.   This encompasses two central themes in his work: the role of technology in society and Andreessen’s inability to understand what people are complaining about.  Let’s say I visit Andreessen’s house and I see a nail gun lying around. It’s a bit of technology which either makes carpenter’s lives simpler or takes work away from them, depending one’s point of view. I wouldn’t care if Andreessen has a nail gun.  If, however, Andreessen was using that nail gun to kill puppies, I would definitely care and Andreessen would apparently assume I have a problem with nail guns.  Dear Marc: it’s not the thing we object to, it’s the abuse. As we read this on a smart phone or a computer, as we ride our bikes to work, as MRIs diagnose diseases and save our lives, few of us are objecting to those things.  Yes, there are certain forms of technology we object to because they’re inherently harmful—internal combustion engines (ICE), for example—but until we have a suitable replacement, no one’s calling for outlawing ICE tomorrow. Even when Europe banned the sale of new ICE after 2035  ,   the time frame was an explicit acknowledgement of the benefits of this technology, while also pointing out the obvious dangers of it.  It’s not technology people object to. Andreessen’s argument is a straw man. He extols the virtues of technology with drooling ecstasy, ignoring that virtue stems from humanity, not tools.  Free Markets  Andreessen writes, “We believe free markets are the most effective way to organize a technological economy.” Two paragraphs later he then rails against central planning.   We believe Hayek’s Knowledge Problem overwhelms any centralized economic system. All actual information is on the edges, in the hands of the people closest to the buyer. The center, abstracted away from both the buyer and the seller, knows nothing. Centralized planning is doomed to fail, the system of production and consumption is too complex. Decentralization harnesses complexity for the benefit of everyone; centralization will starve you to death.   Hayek’s “Knowledge Problem” refers to the famous article The Use of Knowledge in Society  (pdf), by Friedrich Hayek. In this writing, he skewers the notion that central planning can lead to effective economic outcomes. It’s a good, important work and should be read by anyone who wishes a better understanding of the limits of central planning.  Andreessen sets up the great debate, writing “centralized planning is doomed to fail, the system of production and consumption is too complex.” He attacks communism several times, to make it clear that it’s the communist bogeyman he’s warning us about. He also positively cites libertarian works to let us know that laissez-faire is our Savior and Lord.  There are two little problems with him setting up the epic battle of free markets versus central planning. The most obvious is that no one is arguing for central planning. Marx’s ideas, like Marx himself, are dead. There are few economists who take him seriously. Even the Chinese Communist Party has shifted to a “socialist market economy”  , recognizing that central planning doesn’t work. Communism and central planning are safely in the hands of barstool philosophers and pose no threat.  The other problem is that he’s setting up a false dichotomy  . In a tremendously complex world with radically different value systems across its face and mounting climate and ecological chaos, reducing one of the core debates to naïve dualism isn’t enlightening anyone.  However, his writing isn’t always bad.   We believe there is no conflict between capitalist profits and a social welfare system that protects the vulnerable. In fact, they are aligned – the production of markets creates the economic wealth that pays for everything else we want as a society.   I’ve met many a libertarian who argues strongly against welfare on the grounds that people should tend their own estates and government handouts take away our incentive to succeed. Tell that to a single mom working at McDonalds to support two children.   So Kudos to Andreessen for suggesting that social welfare systems are OK.  Curiously, Andreessen later writes, “We believe a Universal Basic Income would turn people into zoo animals to be farmed by the state.” This was stuck in the middle of the manifesto like a porcupine in a petting zoo. I read the text around that several times and I still have no idea what it means, but like most of the rant, it was a bizarre assertion without justification.  Capitalism  While ranting about communism, telling us that only free market capitalism can save us, he blithely ignores the central problem with capitalism today. Yes, he is correct about the power of markets. Prior to capitalism, the world was largely guided by mercantilism  . Capitalism supplanted mercantilism by proving that economics wasn’t a zero-sum game. Eventually, we should hope for a post-capitalist society that addresses the fatal flaw of capitalism: it has no mechanism for handling externalities  .    I don’t know what a successful post-capitalist economic system would look like any more than a mercantilist would have predicted capitalism. Two of the most significant externalities we face today are climate change and ecosystem collapse, but we clearly see that laissez-faire capitalism can’t deal with it effectively. This isn’t just some lefty idea. The US Department of Defense considers climate change a national security priority  The US military isn’t traditionally associated with radicals or lefties.  So Andreessen, by pulling the typical libertarian trick of ignoring the central problem with capitalism, successfully guts his own screed.  Market Discipline  This part is just bizarre. Andreessen writes:   We believe in market discipline. The market naturally disciplines – the seller either learns and changes when the buyer fails to show, or exits the market. When market discipline is absent, there is no limit to how crazy things can get. The motto of every monopoly and cartel, every centralized institution not subject to market discipline: “We don’t care, because we don’t have to.” Markets prevent monopolies and cartels.   I confess that I laughed myself silly at that last sentence.  OK, before we unpack this, we need to understand what “market discipline” is. Per Investopedia  :   Market discipline is the onus on banks, financial institutions, sovereigns, and other major players in the financial industry to conduct business while considering the risks to their stakeholders. Market discipline is a market-based promotion of the transparency and disclosure of the risks associated with a business or entity. It works in concert with regulatory systems to increase the safety and soundness of the market.   There are a couple of interesting things to note about this. First, it talks about risks to stakeholders , nobody else. Risks to customers can become risks to stakeholders, but only if the customers realize that risk. Risks to non-customers, such as from pollution? Who cares? Not their problem. And while Andreessen approvingly quotes Adam Smith about the nature of “self-interest,” there’s another quote of Smith’s which Andreessen didn’t bother to include  (pdf):   People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices.   That, by the way, is from a law review paper entitled “Adam Smith on the Inevitability of Price Fixing.” The quote further reads:   It is impossible indeed to prevent such meetings, by any law which either could be executed, or would be consistent with liberty and justice. But though the law cannot hinder people of the same trade from sometimes assembling together, it ought to do nothing to facilitate such assemblies; much less to render them necessary.   There is no market mechanism to protect against this. The only recourse is in the second interesting thing in Investopedia’s definition of market discipline: regulatory systems. The market’s internal mechanisms for protecting the public have a core problem: businesses are comprised of people and sometimes people think they can get away with stuff and, sometimes, they can. If you decide that Coke, Pepsi, and other products are bad, you can easily switch to water. But what if there is no substitute? Think “insulin.”  And if the market is so great at preventing cartels, why is the US Department of Justice on a tear, prosecuting so many of them  ?  This naïvete about “market discpline” is a symptom of libertarian groupthink.  It’s also worth reading The Market for Lemons  (pdf), an economic paper that won the author a Nobel Prize. It describes how markets degrade when the seller has more information about their product than the buyer. This is called “information asymmetry” and there is, again, nothing in the market mechanism which can guard against this. Even in our “information age,” the buyer often has little way of knowing if the product they’re buying has defects.  Or you can read about the four-square  , an unethical, but perfectly legal technique into pressuring people to pay too much money for a car. Again, the market can’t prevent this. In fact, it incentivizes this unethical behavior to the detriment of the consumer.  Luddites  In addition to railing against the non-existent communist threat, he implies that those who object to our current situation are Luddites:   This upward [techno-capital economic] spiral has been running for hundreds of years, despite continuous howling from Communists and Luddites.   Luddites? In the early 19 th century, the Luddites were  :   British weavers and textile workers who objected to the introduction of mechanised looms and knitting frames. As highly trained artisans, the new machinery posed a threat to their livelihood and after receiving no support from government, they took matters into their own hands.   They were being replaced by low-skilled workers and many of the skilled workers found themselves out of work and starving.  At first blush, this does seem to resemble people like me, worried that my skilled labor is going to be replaced by AI , but this is a faulty comparison.  Eventually, the financial boom created by the Industrial Revolution translated into new jobs and an increased standard of living  . The interim was rough for workers because the government was only acting in the interest of the wealthy and it took decades before those workers found skilled work that the machines could not do.  With how powerful the new AI is in its infancy , where will be the skilled work the AI can’t do? People are already using ChatGPT to help them build entire apps  and plenty of people are losing their jobs to ChatGPT  . As this new technology improves, more jobs will be lost. Unlike the Luddites of old, today’s Luddites are facing a future where there will be few jobs this technology cannot do. When we fully combine it with robotics to replace manual labor, the future looks grim.  What are we to do when we’re losing our homes and can’t afford to eat? Presumably, Andreessen’s answer will be something along the lines of “let them eat cake.”  More likely, since he’s a billionaire, he adheres to the unofficial libertarian motto: “screw you; I got mine.”   Note: because there’s simply too much to cover here, I’ve included an annotated copy of his manifesto here  . Comments are open, so feel free to leave them there or here.  ","title":"Marc Andreessen: Techno Babble","url":"\/blog\/marc-andreessen-techno-babble.html"},{"body":"  Telling Stories    Me and my daughter.   One day, when my daughter was three, she asked me to stop reading stories to her. She wanted me to tell them. So I did. I created a world of faeries and humans and whimsical magic where my daughter would routinely defeat the “evil math faery” by solving a math problem that would break the math faerie’s spell.  She started by counting on her fingers, laboriously working out what one plus two meant. Over the years as the stories evolved, she’s moved on to solving algebra problems, math in different bases ( it’s easier to explain this to younger children ), discovering that the roots of some numbers have have two solutions, and so on. Unsurprisingly, she now excels at math in school.  But she’s getting older. She’s reading books beyond her years, devouring them like a famished dog in a butcher’s dumpster. So I thought it was time to start telling a different story. My intent was to to write a middle-grade novel  , a short work appropriate for those between the ages of eight and twelve. But there were some obstacles.   She often reads beyond her years  It’s taking time and she’s growing older  I can’t seem to help but write “older” material   Thus, my middle-grade novel has morphed into a young adult novel and I’m finding it much more pleasurable to write.  The Reluctant Prince  A classic “many worlds” novel, “The Reluctant Prince” centers around a young boy with autism named “Billy.” When his grandfather is killed and his mother is kidnapped, Billy learns that not only was the man not his grandfather and his mother was keeping secrets, but Billy might be the actual target. A long-lost twin sister, Sia, arrives from Amethyst, the central world. Billy—high school nerd—learns he is probably the heir to the throne of Amethyst...if he can stay alive long enough.  Billy and Sia escape Earth through a slip (a portal to another world) to the underground world of Thonios. Soldiers are already hunting them in this world. There, they’re captured by Hendrix and his nephew, Harris, both soldiers in the Thonian army. The young Harris, who obsesses about his “duty,” is very gung-ho about turning in Billy and Sia as spies, but Hendrix knows that turning the heir-apparent over to a Great House that also claims the throne could lead to war.  Trying to temper his nephew’s militant zeal, Hendrix sits his nephew down with their captives and explains how his wife died while they were stationed on the world of Aarde, guarding a slip to Thonios.  Aarde  Hendrix was tired of the war. It had been years of maneuvering and posturing but little fighting, but that fighting had been fierce and bloody. He and his wife, Alecia, members of the same Quarry, found themselves stationed on the island of Vrede, on the world of Aarde. The entire Quarry had celebrated being posted there.  Aarde was a water\/earth world, a splattering of islands strewn across a deep, blue ocean that spanned the planet. Some islands were huge, but none so large as to be a continent. Strategically the world wasn’t important, but Vrede held a well-known slip to Thonios that had to be guarded.  Hendrix leaned against a palm tree, his feet splayed out on the sandy beach in front of him. Alecia sat across from him. Near the shore, the ocean’s crystal clear waters merged into a deep blue, with fishing boats in the distance and not a cloud in the sky. The air smelled of saltwater.  Hendrix only had eyes for his ‘Lecia. He had taken his shoes off and intertwined his feet with hers. Their commander always gave them leave on the same day. They had been down to the village, a collection of sturdy, one-story brown wooden buildings nestled among the palm trees. Hendrix had never learned the language, but Alecia had thrown herself into it, endearing herself to the locals. She had no trouble negotiating a great price for two bottles of akaat at a large, open-air market. She had become the Quarry’s unofficial ambassador to the Aardans.  Hendrix sipped his akaat, a sweet, runny custard, with just enough alcohol to relax him.  “I don’t know, ‘Lecia. It’s beautiful here, but I’ll take a stone roof any day. Give me a hammer and chisel and a hunk of marble and I’ll carve true beauty for you.”  Alecia smiled and rubbed one of Hendrix’ feet with her own. “I admit the open sky is tough, but I’m getting used to it. I’d never seen a sunset before. And you know how I get when I see a sunset.” Her foot slid up his leg.  Hendrix giggled and a bit of akaat squirted out of his mouth before he could cover it with his hand. He set his drink down and his left hand reached out to her foot. And then his right hand grabbed her other foot and he pulled her closer while she laughed.  An arrow ripped through her throat.   Sia, Billy, and Harris sat there, staring in horror. Hendrix had a haunted look to his eyes, not meeting anyone’s gaze.  “She was in agony, dying in front of me, trying to pull the arrow out. When the second arrow came, it missed me, but I was well-trained. By the gods, I was well-trained. She couldn’t live. I didn’t have my bow. I ran back to warn our Quarry. She saw me run. I let my wife die alone because that was my duty.”  Not Middle-Grade  I didn’t mean to write the above. It just flowed from my fingers as Hendrix spoke. I flinched when I saw the words, but they felt right. That’s when the book morphed from middle-grade to young adult. I’ve had to go back and adjust ages and dialogue as a result.  Note that in the excerpt, the younger Harris wants to scrupulously follow the law and turn in Sia and Billy. It’s the older, perhaps wiser, Hendrix, who is considering breaking the law (and his oath as a soldier). If there is a central theme to the novel, it’s the constant tension between legal and illegal and right and wrong. This is not by accident.  I’m at 63,000 words, but I have little spare time. I’ll return to this precious gift to my daughter someday. Perhaps I’ll have a custom copy printed off for her. Imagine that! A sole copy of a book, written just for you, my angel.  My daughter, however, insists I try to get it published.  One day, perhaps, one day.  ","title":"The Reluctant Prince: An Unfinished Novel","url":"\/blog\/the-reluctant-prince-an-unfinished-novel.html"},{"body":"  I’m on a writing website, Critique Circle  . I want to improve my fiction writing. Periodically they used to post “writing prompts” to give you a chance to practice writing something different.  The prompt was to “create a dialogue between an apprentice trying to learn a unique magic system with a master.” The following is my entry, which took first place.   The mouse sat in the middle of the table, nibbling at the corn.  “Make the mouse disappear.”  The apprentice placed his hands over the mouse and then pulled them back quickly. The mouse was still plainly visible. The apprentice sighed.  The master asked gently, “Why did it fail?”  “Because you didn’t have any reason to believe.”  “Exactly. No one will believe an Illusion of a cliff trying to eat you . . .”  “But they’ll believe an avalanche from that cliff,” completed the apprentice.  There was a sound of glass scraping over stone, followed by a crash. The master’s head whirled around. Seeing nothing, he looked back and the mouse was gone. The apprentice was beaming.  The corn wobbled and a bite appeared. The mouse faded back into view. The apprentice let out a cry of despair and lowered his head into his hands.  “Don’t feel bad,” said his master. “If you had also hidden the corn, the Illusion would have worked. It was quite well done.”  The apprentice sat up a little straighter. “But it’s so hard to always figure out how to create a realistic chain of events on the fly.”  “Practice,” said the master, putting the mouse back in its cage. “Practice.”  He pulled out a heavy dagger and set it on the table. “Now to drive the point home, so to speak.”  The apprentice stared dubiously at the dagger when his master started speaking in low, soothing tones.  “I know this is hard. I know it’s frustrating. You can end all of this pain now if you just gently, gently push the dagger into your throat.”  The apprentice stared at the dagger and, as if it was acting of its own volition, he saw his hand pick it up and slowly start to raise it.  “No!” he shouted, slamming the dagger down on the table. “What the hell was that?”  “Language. Watch your language. And that was a Glamour. It leads you to believe things. Why do you think it didn’t work?”  “Because I didn’t want to kill myself?”  “Exactly. A Glamour, like an Illusion, can only take someone where their belief will already take them. Were you already suicidal, I’d be before the tribunal explaining why I had a dead apprentice.”  The apprentice gulped, realizing just how close to death he had come.  The master patted his hand and said, “I can’t make you do anything you wouldn’t be inclined to already do. Like Illusions, it’s a subtle art. Getting your enemies drunk can help, but I couldn’t, for example, make you want to strangle your mother.”  “You’ve never had an argument with my mother.”  The master burst out laughing. “Fair point!”  The apprentice shifted in his seat, looking at the dagger. “But if magic is so subtle, there’s not much we can really do with it, is there?”  The master reached into his robes and pulled out a deck of cards. “Watch, and learn one of the most powerful tools we have at our disposal.”  He spread the cards on the table, face up. He then scooped them up and flipped them over, face down. And then he started cutting and shuffling the cards, passing them back and forth between his hands in intricate patterns. He fanned them out.  “Pick one, and don’t tell me what it is.”  The apprentice reached for the cards, his hands moving back and forth, and pulled a card out of the deck. It was the three of clubs.  “Will you forget the card you’ve chosen?”  “No.”  The master again shuffled the cards and instructed the apprentice to shove the card back in the deck. More shuffling ensued, followed by placing the cards on the table.  “Turn over the top card.”  The apprentice turned over the three of clubs. He stared in shock. He examined the card closely before turning over the rest of the cards and seeing they were all different. There was no other three of clubs in the deck.  “But,” stammered the apprentice, “how? I know it’s not possible, but I can’t see through your Illusion. That’s not how magic works!”  The master then spent the next half hour explaining false cuts, forcing cards, misdirection, and other ways of deceiving the person watching you with a deck of cards. And then, as if plucking the card from behind the apprentice’s ear, the master held up the three of clubs again. “It takes much practice, but the charlatans in Vegas have mastered the arts of deception. Learn these, convince your enemy that they’re all tricks, and they’ll start believing in your ‘impossible’ Illusions. Even the Vegas charlatans have something to teach us.”  The apprentice sank back in his chair, deep in thought. Finally, he looked up. “The charlatans are paid well for their tricks, yes?”  “The best charlatans are quite wealthy.”  A sly smile appeared on the apprentice’s lips. “We go to Vegas. They already expect the impossible. We’ll be the most famous magicians of all time. We’ll never have to beg for our supper again.”  The master sighed. “I’ve thought of that, but sadly, magic works on minds, not cameras.”  ","title":"Flash Fiction: Charlatans","url":"\/blog\/flash-fiction-charlatans.html"},{"body":"  My eight-year old daughter surprised me again. She likes getting simple math problems to solve at bedtime and for a recent bedtime story, the equation I asked her to solve was this:  $$3\\times\\sqrt{25}\\times5$$  After a pause, she answered 75. I was happy, but then I asked her how she solved it.  For myself, after reducing $\\sqrt{25}$ to 5, I solved 3 x 5 and then 15 x 5.  I then thought 5 x 5 followed by 3 x 25 is easier because multiplying by 25 is easier for me than multiplying by 15. Hence my wanting to know which way she chose to solve it. And her answer?   Three times five is fifteen. So we have fifteen times five. That’s twelve times five, or sixty, and I added fifteen.   Well, damn. Perfectly correct. She’s a reincarnated Babylonian mathematician.  The Babylonians used base 60 math (which they inherited from Sumerians) which is why we have 60 seconds in a minute and 60 minutes in an hour. So naturally my daughter’s comment made me think of the Babylonians and I started doing some reading.  If you’ve ever worked with different mathematical bases, you might be wondering how the Babylonians memorized 60 different symbols for their numbers.  Turns out they didn’t. They only had two symbols, a unit symbol and a 10 symbol (so, some hints of base 10). We can’t easily write cuneiform, but if we pretend the units symbol is | (the pipe character) and the tens is «, they we could write 54 as «««««||||.  In Unicode’s interpretation of cuneiform (assuming you have the font installed), that’s 𒐐𒐘. But note that cuneiform is complicated and evolved over millennia, so the Unicode representations don’t appear to quite match the real ones I’ve seen online.  But where does base 60 come in? Well, recall that in base 10, the number 365 can be thought of as this:  $$3\\times10^2+6\\times10^1+5\\times10^0$$  For the Babylonians, they might have written |||||| ||||| (a six and a five with a space in between) which is:  $$6\\times60^1+5\\times60^0$$  Or in cuneiform, 𒐚 𒐙, or after 300 CE: 𒐚𒑱𒐙.  You might ask how they dealt with zero. For example, the number 1 (one) might be 𒐕, but the number 60 (1x60) would also be 𒐕 unless there was a space after it to denote the units position: 𒐕𒑱. Apparently the Babylonians didn’t think this a problem and, in fact, that middle '𒑱' character to separate digits was only introduced about 300 CE, so prior to that, you’d have to pay very careful attention to the position of the numbers in columns to hope you interpreted ambiguous numbers correctly.  That separator character solves all sorts of grief, I might add. The number 61 is 𒐕𒐕, but the number 2 is 𒐖. It’s easy to get them wrong unless you write 𒐕𒑱𒐕.  That being said, after mentioning to my daughter that Babylonian numbers were base 60, she immediately wanted to know about them. So I showed her the basics and after converting our year, 2019, to Babylonian, we got 𒌍𒐗 𒌍𒐝 (33;39). I then worked with her to show her that in different bases, the “columns” of numbers of powers of the base (from right to left, they’re 0,1,2,etc.), so after she translated the above “digits” to 33 and 39 respectively, I showed her how that works out to $33\\times60^1 + 39\\times60^0$. And that equals 2019.  Her mind was blown, but she thought it was one of the coolest things she’s ever seen.  You can read more about Babylonian numbers here.   It should also go without saying that I’m hardly an expert on cuneiform of Babylonian mathematics, so if you or or know anyone who is, I’d love corrections.  And if you’re software developer, here’s a quick Perl script I wrote to convert base 10 numbers to Babylonian:  #!\/usr\/bin\/env perl use strict; use warnings; use utf8; binmode STDOUT, ':utf8_strict'; foreach my $num (@ARGV) { my $orig = $num; if ( $num &lt; 1 || $num != int($num) ) { die &quot;'$num' is not a positive integer&quot;; } my $base = 60; my $space = '𒑱'; my %numbers = qw( 0 ␣ 1 𒐕 2 𒐖 3 𒐗 4 𒐘 5 𒐙 6 𒐚 7 𒐛 8 𒐜 9 𒐝 10 𒌋␣ 11 𒌋𒐕 12 𒌋𒐖 13 𒌋𒐗 14 𒌋𒐘 15 𒌋𒐙 16 𒌋𒐚 17 𒌋𒐛 18 𒌋𒐜 19 𒌋𒐝 20 𒎙␣ 21 𒎙𒐕 22 𒎙𒐖 23 𒎙𒐗 24 𒎙𒐘 25 𒎙𒐙 26 𒎙𒐚 27 𒎙𒐛 28 𒎙𒐜 29 𒎙𒐝 30 𒌍␣ 31 𒌍𒐕 32 𒌍𒐖 33 𒌍𒐗 34 𒌍𒐘 35 𒌍𒐙 36 𒌍𒐚 37 𒌍𒐛 38 𒌍𒐜 39 𒌍𒐝 40 𒐏␣ 41 𒐏𒐕 42 𒐏𒐖 43 𒐏𒐗 44 𒐏𒐘 45 𒐏𒐙 46 𒐏𒐚 47 𒐏𒐛 48 𒐏𒐜 49 𒐏𒐝 50 𒐐␣ 51 𒐐𒐕 52 𒐐𒐖 53 𒐐𒐗 54 𒐐𒐘 55 𒐐𒐙 56 𒐐𒐚 57 𒐐𒐛 58 𒐐𒐜 59 𒐐𒐝 ); my @result; do { unshift @result =&gt; $num % $base; $num = int( $num \/ $base ); } while ( $num &gt;= $base ); unshift @result =&gt; $num if $num; my $babylonian = join $space =&gt; map { $numbers{$_} } @result; print &quot;$orig is '$babylonian'\\n&quot;; } __END__ =encoding utf8 =head1 NAME babylonian.pl - print Babylonian numbers =head1 SYNOPSIS $ .\/babylonian.pl 123456789 𒐝𒑱𒌍𒐕𒑱𒌍𒐗𒑱𒌍𒐗𒑱𒐝 =head1 DESCRIPTION Given a positive integer, prints the Babylonian version of it. See also: https:\/\/mathandinformatic.wordpress.com\/numbers-and-number-theory\/  ","title":"Babylonian Numbers for 8-Year Olds","url":"\/blog\/babylonian-numbers-for-8-year-olds.html"},{"body":"  In the US, I have lived in Texas, Louisiana, Washington state, Oregon, Alaska, and Hawaii. I’ve also lived in Japan, the UK, the Netherlands, and now France. I grew up in serious poverty, I have been homeless, and I have seen the world. There’s nothing like experiencing other cultures first-hand and understanding what a weird, wonderful bunch humanity is.  London, United Kingdom, 2007  The following is from memory, not an old post, so some details might be off.    The River Thames with the Houses of Parliament Source    My brother Greg and his friend Tom had bought tickets to introduce me to football, the most popular sport in the world. It was for a minor team I had never heard of. Despite the minor team, there was a major police presence. Hoolaganism was the order of the day and the police were having none of it. I was warned not to stray to close to the fans on “the other side” and, indeed, the crowds were restless. As we were leaving the match, the police were making their arrests. The British take this sport a wee bit too seriously.    One of my brothers, Greg.   Greg, Tom, and I are making our way back to the Tube, I glance in a doorway and see a scene that could have been out of an art film. I stop and let Greg and Tom walk ahead as I take in the view. A lovely pub courtyard, with an old man nursing a pint lecturing a young boy. Could have been his son. Or grandson.  The man was warning the boy about the dangers of the world, the [insert list of racial slurs here] ruining England, and how everything is going to hell.  We all have hills we’re willing to die on and my hill is named “Fuck Bigots.”  Greg and Tom are far ahead of me now, but I march into the courtyard, straight to the table with the old man and the boy. Like committing a crime of passion, I have a reaction, not a plan. The courtyard is deserted save for the two at the table staring at me in surprise. I ignore the man and look at the boy. I am alone and this man looks like he has had led a rough life. This is not an intelligent thing to do.  I say, “I’ve traveled to and lived in many countries and there are wonderful, beautiful people everywhere. The world is an amazing place. Don’t let this man’s hatred poison you.”  The look of surprise on their faces turns to shock. Neither of them say a word. I wonder if my American accent, nice clothes, or apparent lack of fear caught them off guard. No one follows me as I turn and walk away, catching up with Greg and Tom.  I still think about that boy, wondering if he understood. I wonder if that boy remembers me. I hope he’s doing well and gotten beyond an old man’s hate.  Brussels, Belgium, 2017    Brussels, Belgium Source    February 3rd, 2017. I am in Brussels to speak at a conference. I hail a taxi at the Brussels airport and Leonid Brezhnev is behind the wheel. Or at least someone who looks like him.  Taxi drivers are a mixed lot. Some like to talk. Some don’t. Mine does. He was born in Belgium in a small town near the German border in the 1930s. I ask him how he came to be living in Brussels.  Back in 1958, he went to Antwerp to study at the Colonial School  . After six months of study, he was sent to Burundi to help oversee their colony. Shortly after his arrival a new doctor arrived and, the moment he saw her, he immediately decided to be sick. Two of their six children were born in Burundi. They left for Brussels in 1962 when Burundi regained their independence.  He was so inspired by his wife’s career that he returned to university and earned a Master’s Degree in business administration. He went on to be the general manager of a large company and after several years in that role, he was unceremoniously fired when a German company bought it. He had a hard time finding work at his former level and refused to “lower himself,” despite his children urging him to take anything he could find.  And then his wife died and he had no choice. He became a taxi driver, but was always filled with shame when he would see a business person he used to work with. He’d hide his face and hope they wouldn’t see him.  On that day he is 81 years old and the shame is long gone. He drives two days a week just for something to do and only takes airport passengers because they’re much more interesting to talk to than fares around town.  Grasse, France, 2022    Grasse, France (own work)   I have dropped off our car to have the brake pads changed. As I wait for an Uber on a deserted side street, a long-haired blond gentleman walks up to me, all smiles. I am wary.  (All conversation is in French)   Him: Lovely day!  Me: Yes, it is. Not a cloud in the sky.  Him: Ah, you’re English!  Me: American, actually.  Him: You must be from Los Angeles.   &lt;blink&gt;   Me: Texas, actually.  Him: You’re from Houston!  Me: No.  Him: You’re from near Houston?   Am I? Hard to say. What’s “near” in this context?   Me: Actually, no.   I’ve had weird conversations like this before. Usually they’re from someone eager to chat with a foreigner, pleased that they can communicate. However, from time to time, you experience this in tourist areas from people looking to take advantage of tourists with too little sense and too much money. They quickly shift to suggesting places to visit, offer to carry your bags, or maybe recommend a “friend” who can give you a cheap lift. Caveat emptor.  But that can’t be the case. I’m in Grasse, a small town in the French countryside. Tourists don’t go out of their way to visit Grasse (though they really should).   Him: Are you a tourist?  Me: No. I live here.  Him: OK. Bye.   And he walks away.  Leipzig, Germany, 2022    Leipzig, Germany Source    Later that week I arrive at the Nice airport in southeast France. I am very early, as I tend to do. I was heading to Leipzig, Germany, to speak at another conference. I’m grateful I arrived early because while there are no lines for other airlines, Lufthansa doesn’t have any quick check-in kiosks and it takes me over an hour to check in. While I’m in line, the lady behind me starts chatting about the line and the wait. Small talk.  Later, as I’m boarding the first leg of my flight to Frankfurt, I find I’m sitting behind her. We talk some more and she’s going to Leipzig, too.  At the Frankfurt airport, we pass the time having a quick drink and it turns out that she retired as a Russian ballerina after 22 years. She speaks seven languages and is learning to fly a private jet as part of her new job. She chats with everyone around and effortlessly switches between German, French, English, and Russian. I take her word on the Finnish, Italian, and Estonian.  During conversation I mention that my father, Jim Poe , lived in Moscow in the 1970s and I hadn’t been able to get more information about his time there. She replies that she has contacts and she’d love to help. We swap business cards.    We eventually get to Leipzig. She asks if I have a ride and I explain that my ride just sent an email apologizing for having to cancel. She insists we split a cab, but as she exits, she pays the entire fare, telling me not to worry about it.  She leaves the cab and tips the driver handsomely. As he is driving me to my hotel, already having been paid, he decides to take some side streets to show me some of the magnificent architecture in Leipzig. He is very enthusiastic about it. I have no idea what he’s saying. He points out buildings left and right. He doesn’t speak more than a couple of words of English. I suspect my accidental sightseeing tour is fantastic, but his enthusiasm makes up for the lack of understanding.  Even though I don’t have to pay, I offer a tip which he refuses. He hands me my bags with a smile and with a hearty and happy something or other , he waves and drives away.  Leipzig, Germany, the next day  I am downstairs in the hotel restaurant the next morning, searching for coffee for my room. Apparently I can’t have a pot delivered, but I can get a cup and carry it up. A young lady sitting at a table calls out to me.   Her: Excuse me! Sir!  Me: Yes?  Her: Can you help me?  Me: Uh, sure. I’d be happy to.   She stares at me. I stare back. Awkward silence. I’ve missed something.  An NPC laughs in the background. The lady looks at the NPC and turns back to me, blushing.   Her: I mean, can I help you?   I notice the pin on her lapel. She works here.   Me: I want to get some coffee for my room.  Her: You have to pay.  Me: That’s fine.   Someone leans over and whispers in her ear.   Her: Never mind. Go ahead.   So I do. The coffee machine doesn’t work.  Châteauneuf Grasse, France, Now  As I write this, my daughter is sitting on the couch, reading. My wife is working hard. I’m only 55 years old, but I’ve lost much of my hearing and my sense of taste due to surgeries. I should have them back within the year, pending more surgeries. Today is also the birthday of someone I used to know and I only learned about her passing after doing some digging when I noticed she stopped posting to Facebook.  Time is passing and I’m keenly aware of it. I’m not old, per se , but I’m getting older. My beard is grey and the hair on the top of my head is just starting to follow. Within a two or three decades, I’ll probably be gone and these words will be forgotten, along with my life. Is it narcissism that leads me to write these entries? I like to think that I’ll be leaving something for my daughter to remember me by. Immortality by proxy. It’s why I write my personal stories .    A small statuette my great grandmother bought in South America   There is, on my desk, a small figurine my great-grandmother Atwood picked up in her travels to South America. It was handed down to my grandmother. Not having a great relationship with my mother, my grandmother gave the statue to me before she died. I’ve told my daughter that it’s hers. An immortality by proxy for her great, great grandmother Atwood, though in name only. I have no stories for her.  All these stories lost forever. All of these wonderful, confusing people spending a brief moment of eternity with one another.  ","title":"Adventures in Traveling","url":"\/blog\/adventures-in-traveling.html"},{"body":"   “Hit the skeleton with the sword, take his ring, and then walk north and touch the altar.”   The year is 2022 and 35 years later, I still remember that damned sentence. The year was 1987 and I was writing a text adventure in BASIC for the COCO 2  .    The TRS-80 Color Computer 2 Source    Because it included BASIC for free, and because that was the only programming language I knew at the time, that’s what I was coding it in. If you’ve never seen BASIC, here’s a hi-lo game programmed in it  :  10 PRINT TAB(34);&quot;HI LO&quot; 20 PRINT TAB(15);&quot;CREATIVE COMPUTING MORRISTOWN, NEW JERSEY&quot; 30 PRINT:PRINT:PRINT 100 PRINT &quot;THIS IS THE GAME OF HI LO.&quot;:PRINT 110 PRINT &quot;YOU WILL HAVE 6 TRIES TO GUESS THE AMOUNT OF MONEY IN THE&quot; 120 PRINT &quot;HI LO JACKPOT, WHICH IS BETWEEN 1 AND 100 DOLLARS. IF YOU&quot; 130 PRINT &quot;GUESS THE AMOUNT, YOU WIN ALL THE MONEY IN THE JACKPOT!&quot; 140 PRINT &quot;THEN YOU GET ANOTHER CHANCE TO WIN MORE MONEY. HOWEVER,&quot; 150 PRINT &quot;IF YOU DO NOT GUESS THE AMOUNT, THE GAME ENDS.&quot;:PRINT 160 R=0 170 B=0:PRINT 180 Y=INT(100*RND(1)) 200 PRINT &quot;YOUR GUESS&quot;; 210 INPUT A 220 B=B+1 230 IF A=Y THEN 300 240 IF A&gt;Y THEN 270 250 PRINT &quot;YOUR GUESS IS TOO LOW.&quot;:GOTO 280 270 PRINT &quot;YOUR GUESS IS TOO HIGH.&quot; 280 PRINT:IF B&lt;6 THEN 200 290 PRINT &quot;YOU BLEW IT...TOO BAD...THE NUMBER WAS&quot;;Y 295 R=0:GOTO 350 300 PRINT &quot;GOT IT!!!!!!!!!! YOU WIN&quot;;Y;&quot;DOLLARS.&quot; 310 R=R+Y 320 PRINT &quot;YOUR TOTAL WINNINGS ARE NOW&quot;;R;&quot;DOLLARS.&quot; 350 PRINT:PRINT &quot;PLAY AGAIN (YES OR NO)&quot;; 360 INPUT A$:IF A$=&quot;YES&quot; THEN 170 380 PRINT:PRINT &quot;SO LONG. HOPE YOU ENJOYED YOURSELF!!!&quot; 390 END  Not too complex, but you’ll notice a couple of easy-to-miss GOTO statement which make flow control a touch odd. Still, that’s what we had to work with.  What’s worse is that BASIC was an interpreted language. Today, many languages are compiled directly to machine code, such as C. Some languages, like Perl, are compiled to byte code before they’re executed. BASIC was interpreted and executed line by line. If you had a 1,000 line BASIC program and a syntax error on line 793, you often wouldn’t find out unless that line executed. It made development hard , but it did mean that we were constantly manually doing QA on our programs, something that is often skipped in today’s TDD world.  And if that’s not bad enough, trying to optimize BASIC programs often meant shortening variable names so that they would take up less memory!  Ignoring my whining about how hard life was back then, for my text adventure, I wanted a sentence parser that was better than Zork, Adventure, and other games, but before I could parse the sentence, I had to lex it (break it into tokens). In the world of BASIC, this was no easy task.  When writing a parser, you first lex the data into discrete tokens (words, in our case) and parsers take those words, apply a grammar, and figure out what the sentence is (or if it is grammatically incorrect).  Naturally, the 1987 me had no idea what that meant. I just knew that I needed to convert those words into numbers that I could use with an ON X GOSUB command to dispatch to the correct subroutine. The subroutine would read subsequent words off of the stack to figure out what to do. I’d figure out the hard stuff later.  But the hard stuff was lexing the string into tokens.   “Hit the skeleton with the sword, take his ring, and then walk north and touch the altar.”   When I wrote my lexer, it worked flawlessly. But it took eight seconds to lex that string into the proper numbers.  Eight seconds. I remember that very distinctly because it was at that moment that I knew my dreams of writing a grand text adventure were dead if I couldn’t get that working. All of my work mapping out the hotel you woke up in and the puzzles you would have to solve were for nothing if I couldn’t fix this.  So I fixed it.  I knew that machine code would run much faster than BASIC, though I didn’t know how much faster it might be. I was just assuming it would improve the performance.  My roommate had an assembler program for the computer, along with a manual. I pored over them and because BASIC has such a paucity of features, it was not too difficult to convert the lexer to assembler, though it took me a few days.  But I had to call this routine from BASIC. That’s an odd combination of CLEAR (reserve memory), DATA (define your program), POKE (put your program in memory), DEFUSR... (define the user routine used to call your program) and USR... (call the user routine) commands. Here’s a small program to print out the ASCII value of whatever character the user typed ( from the user manual  ) (warning: pdf):  5 CLEAR 25. 12000 'RESERVE MEMORY 10 DEFUSR1=1200015 CLS 20 FOR I = 1 TO 28 &quot;STORE EACH BYTE OF OBJECT CODE 30 READ B: POKE 12000 + I. B 40 NEXT I 45 'HERE IS THE OBJECT CODE 50 DATA 173,159,160.0 60 DATA 39.250.128.10.38.12 70 DATA 173.159,160.0.39.250 75 DATA 129. 65, 45. 2 80 DATA 128. 64, 31.137,78 90 DATA 126.180. 21111 99 'TELL BASIC WHERE THE ROUTINE IS 100 POKE 275. 15: POKE 276. 211 110 A = USR1(0) 'CALL THE SUBROUTINE AND GIVE RESULT TO A 115 IF A =13 THEN END 120 PRINT &quot;CODE =&quot;; A 130 GOTO 110  Doing this for my assembly code was daunting, but I got it working. And it ran blindingly fast! I went from eight seconds to only the slightest of pauses. It was incredible! It was a miracle! It was wrong! It was giving me the wrong numbers.  I printed out the assembler on the dot matrix printer and laid out probably eight to ten meters of paper (my memory is probably faulty), got on my hands and knees with a pen and started laboriously tracing through that code, line by line. This was called a desk check  and was one of our primary debugging techniques in the 80s.  Hours later, with sore knees (and with my roommate claiming I was showing off), I found my bug. I fixed the program, reran it, and it was a success. I had a lexer for simple English sentences and could move on to my parser.   I never finished that game. It was my roommate’s computer and he moved away. By this time I was learning C at Lower Columbia College  , Except that there was a big problem. This was the first C class offered at LCC and we didn’t have a C compiler. We had a Prime Minicomputer  we could use, but Garth Conboy  , who was writing the compiler, hadn’t finished it yet. So our first few C programs for class were written out in long-hand and submitted to our professor who would manually check them for correctness (desk checks again!).  I still remember the day that Professor Little brought in his computer setup and demonstrated one of the programs his company produced. It seemed awesome. Someone asked him about the sales of the program and Professor Little started to explain that it was very popular. It was so popular, in fact, that pirates were copying his program and redistributing it for free.  And then our professor broke down crying in front of all of us. He was teaching at LCC because so many people were using free copies of his software instead of paying for it that his company was going broke. It was a humbling experience for all of us to watch our professor cry.  Later, when we got the C compiler, the first version was buggy. Some expected commands didn’t work and, one day, while debugging a problem, I added a printf statement and the program worked! I eventually just printed an empty string and that was enough to get my program to run. I had to show it to my professor and explain the problem to ensure he wouldn’t mark me off on it.   Fast forward to 2022, 35 years later, and I still think about that BASIC program. Eight seconds to lex that string. I had gotten to the point where I could walk about a map, pick up and drop items, but that was all. But damn, eight seconds! And between me being inexperienced (I had only started programming in ‘82), and having to use an assembler, it took me two weeks (as I recall) to get the lexer working at an acceptable speed.  What would it take today? The following is a rough hack, but it shows (sort of) what I was doing back in 1987.  #!\/usr\/bin\/env perl # Less::Boilerplate is a personal module I use for quick # hacks, but it makes my Perl more &quot;modern&quot;. You can read that as # &quot;use less boilerplate&quot; or &quot;useless boilerplate&quot;. Both are correct. use Less::Boilerplate; use Benchmark 'countit'; my $words = get_words('\/usr\/share\/dict\/words'); my $sentence = 'Hit the skeleton with the sword, take his ring and then walk north and touch the altar.'; say countit( 8, sub { parse_sentence( $sentence, $words ) } )-&gt;iters; sub parse_sentence ( $sentence, $lookup ) { # this is not perfect, but it replicates when I did back in the 80s. my @words = split \/\\s+\/, lc $sentence =~ s\/[[:punct:]]\/\/gr; my %numbers; foreach my $word (@words) { if ( my $number = $lookup-&gt;{$word} ) { $numbers{$word} = $number; } else { warn &quot;$word not found&quot;; } } return \\%numbers; } sub get_words ($file) { open my $fh, '&lt;', $file; my %words = map { state $i = 1; lc($_) =&gt; $i++ } split \/\\n\/, do { local $\/; &lt;$fh&gt; }; return \\%words; }  That consistently prints out a number around 1.9 million. In other words, this quick ‘n dirty hack that I wrote in a couple of minutes replaced two weeks worth of work in 1987 and ran almost two million times faster.  That’s right. This trivial example was not only easier to write, but it was two million times faster than the BASIC code I wrote 35 years ago.  You kids have no idea how easy you have it.  ","title":"Programming in 1987 Versus Today","url":"\/blog\/programming-in-1987-versus-today.html"},{"body":"  If you have an interest in math, by now you may have hear that the mathematician Po-Shen Loh has found an easy way to solve quadratic equations. Those equations looked like this:  $$ax^2 + bx + c = 0$$  And they were every student’s nightmare in algebra class, because when factoring the equation didn’t work, you learned this solution:  $$x={-b\\pm\\sqrt{b^2-4ac} \\over 2a}$$  Finding the solution for many quadratic equations was a difficult chore, especially if the answers involved imaginary numbers. Thanks to Loh, it’s now easy. Unfortunately, many of the sites I’ve visited seem to have rather cumbersome descriptions of this relatively easy solution. So to help myself remember the solution, and thus help you learn the solution, let me skip the proof of this novel solution  and just show you how to solve it. If you wish to know how it works, there are many sites on the web which will get into the details. Just hit your favorite search engine for something like “Po-Shen Loh quadratic equation”.  You’ll only need basic algebra to understand this.  Step 1: Getting x 2  Loh’s solution requires that the first term, $x^2$, is always just $x^2$, and never multiplied or divided by anything. So we don’t want the first term to be $3x^2$ or $x^2 \\over 4$.  That’s fine, because we can just multiply or divide, as needed, so get to $x^2$. For example, if we start with this equation:  $${x^2 \\over 2} - 3.5x + 6 = 0$$  We can just multiply by $2$ to get:  $$x^2 - 7x + 12 = 0$$  Step 2: Getting $-b$ and $c$  Next, we need to isolate the opposite of $b$. Since quadratic equations are in the form $ax^2 + bx + c = 0$, and we have $x^2 - 7x + 12 = 0$, we know that the opposite of $b$ is $7$ (since it’s $-7$ in the equation).  Next, we need $c$. In the equation $x^2 - 7x + 12 = 0$, we can see that $c = 12$.  The key insight in Loh’s work is that we’re searching for numbers which add to $-b$ and multiply to $c$. Thus, for our example:   Sum ($-b$) is $7$  Product ($c$) is $12$   If you remember the above, you’ll remember this in no time because that’s the key to making everything work.  Step 3: Finding the roots  So we need two numbers which add to $7$ and multiply to $12$. Rather than guessing, we start with the addition.  First, we know that two numbers, added together, will equal any number if their average is half of that number. So let’s take half of that $7$, or $3.5$ (or $3{1 \\over 2}$ if you prefer fractions).  So we know that the two numbers are (using $u$ for a placeholder) are $3.5 - u$ and $3.5 + u$.  Since multiplying them gives us $c$, or $12$, we have:  $$(3.5 - u)(3.5 + u) = 12$$  Using the difference of squares, left hand side becomes $3.5^2 - u^2$, or $12.25 - u^2$. So we have:  $$12.25 - u^2 = 12$$  And that gives us:  $$.25 = u^2$$.  Taking the square root of each side gives us $.5 = u$. Since we that the numbers are $3.5 - u$ and $3.5 + u$ we know that the solutions are 3 and 4.  You can substitute those numbers into this equation to check the answers:  $${x^2 \\over 2} - 3.5x + 6 = 0$$  Simplifying  Skipping over many mathematical steps, let’s take this:  $$4x^2 - 8x - 16 = 0$$  We want $x^2$ as the first term, so divide each side by four: $x^2 - 2x - 4$.  We know that $-b$ is $2$ an $c$ is $-4$, so we have $(1 - u)$ multiplied by $(1 + u)$ = $-4$.  $$1 - u^2 = -4$$ $$5 = u^2$$ $$\\sqrt{5} = u$$  So the roots are $1 \\pm \\sqrt{5}$.  Let’s do it again  Let’s use this equation:  $$x^2 - 2x - 24 = 0$$  We know that $-b$ is $2$ an $c$ is $-24$.  So half of $-b$ is $1$, we have:  $$(1 - u)(1 + u) = 24$$  That reduces to:  $$1 - u^2 = 24$$  Which becomes:  $$25 = u^2$$  So we can see that $u$ is $5$ and $(1 - u)$ is $-4$ and $1 + u$ is $6$, giving us the roots of the equation. See how simple that is? It takes practice, though.  A harder example  Let’s take this equation:  $${x^2 \\over 2} - x + 12 = 0$$  We need $x^2$, so let’s multiply both sides by $2$, giving us:  $$x^2 - 2x + 24 = 0$$  We know that $-b$ is $2$ an $c$ is $24$.  So half of $-b$ is $1$, we have:  $$(1 - u)(1 + u) = 24$$  That reduces to:  $$1 - u^2 = 24$$  Which becomes:  $$-23 = u^2$$  So $u$ is $\\sqrt{-23}$, or $i\\sqrt{23}$.  Thus, the two possible solutions are $1 \\pm i\\sqrt{23}$.  That’s not a fun problem to solve with the traditional quadratic equation solution.  If you want more practice watch Loh’s video on the topic.     ","title":"The Easy Solution to Quadratic Equations","url":"\/blog\/the-easy-solution-to-quadratic-equations.html"},{"body":"  My Muse is back. Unfortunately, she’s not a very good one. I imagine her sitting on a barstool with too much makeup, chipped nail polish and an unlit cigarette dangling from her lips. She bats asterisks of mascara at me and if I light her cigarette she’ll push a pen in my hand. It seems she’s only good for one-night stands or at best a weekend fling. Period.  And what would I write? Maybe I’d pen “Emma”, but I’d make her a prostitute and ruin the story. Or “Romeo and Julian”, the Capulets and Montagues sitting around, fast friends, with wife-beaters crawling up their beer-engorged guts and muttering that them faggots ain’t natural.  My Muse would be aghast, her chins quivering in disgust.  And the fruit of my labor? Two published books, both on computer programming. My first article in print? “Logic Programming in Perl,” published in the now defunct The Perl Review. Numerous technical articles on the web. I love what I do for a living, but my Muse keeps sneaking back and pinching my bottom for another flirtation with writing fiction.  Don’t get my wrong. I do write fiction. I wrote a screenplay, have an unfinished screenplay, an unfinished novel. A few short stories. All of it unpublished.  I fear my long-standing fling with my Muse has conditioned me to accept what I do have rather than what I could have. I mock her, but she knows how to show me a good time if I only let my guard down. She’s an escritorial fuck buddy I’m unwilling to introduce to my friends.  I think of writing as a dream. I also think of it as a supreme act of narcissism. Of vanity. Who am I to think that others will want to read what I write? And yet here I am, perversely, writing. I’ve stolen a quickie from my Muse. No wonder she doesn’t stay long. Why should she satisfy me if I don’t satisfy her?  Sometimes we go for long walks together and we plot our novel. How is this different from daydreaming? Writing is almost respectable, but until I set pen to paper or fingers to keys, even toying with respectability eludes me.  I’m a daydreamer who has succeeded by finding himself competent in a field he enjoys. And it’s not even that I’m the best in my field. I’ve worked hard to learn more, but harder to market myself. I get hired not because I am an astonishingly good programmer. I get hired because I am Ovid. Because my ability to write, though not noteworthy, is nonetheless more interesting than a few others in my field and people remember me.  I am known for my writing, not my technical ability. Despite this I rely on the latter and not the former. I know if I pay attention to my Muse, perhaps even trouble to learn her name, she’ll reward me. She’ll strip off her makeup, her fickleness, and open herself to me. Perhaps I’d be poorer and happier and writing about missing the technical challenge of programming, but at least I would know.  Until then, my writing will reflect my Muse. Sickly sweet perfume and trying too hard to please. My writing, though cloying, reflects how I think when the Muse is nearby. I’m a saccharine Faulkner. I’m artificial. My Muse and I grunt and groan and sweat and thrust and play and writhe and finally lie back in exhaustion, each secretly fearing that we’ve let the other down. And I still don’t know her name.  ","title":"Why I Write","url":"\/blog\/why-i-write.html"},{"body":"    Early Life  The Soviet Years  Valerie  Brenda  The Children  Credits  Update 1  Update 2     This is a eulogy for my father because his story should be told. Much of this story is rambling and hard to follow. This is because I’m grieving for the father I never had and because no one in our family knows what happened in this man’s strange life. If you’re confused, so are we.  And if you have any stories to share of Jim’s life, please share them in the comments. Corrections are also welcome because I’m quite certain that some of this is dead wrong.    James Lewis Poe, April 22nd, 1941 -June 26th, 2019. RIP.   My father, Jim Poe, has passed away. He died on the morning of June 26th, 2019 and this is the story of his life as I know it. I’ll try to be fair, but there are several unreliable narrators in this story, not the least of whom is me. Further, most dates in this story should be taken with a grain of salt because it’s very hard to piece them together. All of Jim’s known surviving children, and his fourth wife, Valerie, have contributed to this recollection.  Early Life  This writing is an attempt to process my grief for a man I did not know, but who I always wanted to have as part of my life. And I have, to my astonishment, a plethora of heretofore unknown brothers and sisters who are similarly grieving. To all of his children who’ve survived Jim (Gayle, Greg, Lewis, Lynne and possibly others we don’t know about), I love you dearly and forgive me for airing dirty laundry. And for all our father’s lies, I do believe him when he denies being a Soviet spy. More on that later.    Jim and my wife, Leïla.   Jim was born in Texas in 1941.   He has a surviving brother who lives in Colorado.  When Jim was 16 years old (though given his numerous “birth dates”, this is questionable), he married for the first time, to a 13-year old girl. This was legal at the time and given that she was with child, understandable. Their child, Randy, was my half-brother. I first learned about his existence when I was about 12 or 13, when my mother, Jim’s second wife, came outside to tell me that Randy was dead. Apparently he went into the woods with a gun and never came out. Hunting accident? Suicide? Murder? I don’t know. I just know that, at that age, I was confused I had never been told I had another brother.    Jim Poe, December, 1965   After Jim had left his first wife, he talked wistfully about a woman named Janis who he briefly dated in Texas. Jim being Jim, they drifted apart and over the years, he said he wondered what happened to her. In 1967, the year I was born, he found out. Janis Joplin  became a superstar. That being said, Jim is one of the many unreliable narrators of this story.  Jim met my mother, Caroline “Karee” Tom around 1961, in Portland, Oregon. They fell in love and had a quick, private wedding, much to the consternation of my maternal grandparents. In 1962, my sister Gayle was born. A couple of years later, our brother, James Lewis Poe Jr., was born, but he passed away at six months, due to a heart problem from birth. I was born in June of 1967, in Fort Worth, Texas, the town our father had fled from.  I use the word “fled” with care because I don’t know exactly what happened. Jim told me what happened, but he was very drunk at the time and later denied all of it. Other family members have shared variations of the story, but with it all being hearsay and drunken recollections, I don’t want to tar his name with a story I don’t know to really be true. I do remember, however, Jim being sober and telling me he joined the navy “one step ahead of the boys in blue” (the police). That would fit with every variation I’ve heard.  In the early 60s, Jim was a sailor and was briefly in Portland, Oregon, where, as mentioned, he met my mother. He had already left his first wife and my mother dropped out of college, at least in part to be with him. They traveled quite a bit and in 1966, while he was in the Navy stationed in Gourock, Scotland, I was conceived. My sister Lynne, née Charlotte, was also conceived in Scotland to a different mother, Eliza.  Lynne was conceived two weeks later than I. I like to describe it as an embarrassing exercise in familial mathematics. Lynne just says we’re twins. I think Lynne’s description is nicer.  Our brother Greg, same mother as Lynne, was born February, 1969. And later our brother Lewis was born April, 1978, to Valerie Poe, Jim’s fourth wife (between Eliza and Valerie was Janath, but I don’t know much about her).   If you’ve been keeping track that makes seven children by four mothers. That we know of. Jim denied there were any more, but given that he’s often been less than honest with us, we’re skeptical.    Our sister, Gayle, was the only sibling I knew growing up. She remembers Jim as a wonderful father who would sometimes have strange people visiting our home in Scotland, including a man with one hand. My paternal grandmother, Birdie (who I didn’t know, but Gayle did), recollects that Jim had a solid career in the Navy and had received a commendation for helping to capture a Soviet spy onboard ship. This was the first of many strange “Soviet” stories surrounding our father. However, I can’t verify this because even though I filed a Freedom of Information request for Jim’s naval records, I was informed the US Navy had no records for him.  That being said, Gayle shared this tidbit:  I remember him picking me up like a sack of potatoes and putting me over his shoulder when he would come home. I don’t like peppermint lifesavers, but I loved them because he gave them to me.  She also said:  I remember dad coming home one night and a man was with him. He had a bandage on the end of one arm. I, being 4 or 5, thought he had lost a hand. I also remember mom throwing a metal file box at him and screaming at him. They left. I questioned him about that when I was staying with him in Idstein (Germany). He remembered that but said I was mistaken about the missing hand.    My father’s ship, USS Simon Lake, Holy Loch, Scotland, 1969   It was during this time in Scotland that the relationship between Jim and my mother deteriorated, as evidenced by my sister Lynne’s birth two weeks after my own. There’s much more I can say, but I don’t know what is true and what is not.  My mother returned to the US (possibly involuntarily) and I was born in Fort Worth. Jim left the Navy at this time and applied for many jobs in the UK, hoping to stay. But he had to return to the US and when he first saw me in early 1968, he picked me up and I vomited all over him. I was a precocious child.  As Jim tells the story, he was in the US, but quickly received a job offer in the UK and he was on the next plane back, never to live in the US again. I was told at one point that mom had been contacted by the FBI regarding Jim’s whereabouts, with instructions to contact them if he ever returned to the US, but I’ve no idea if this is true. Many years later Jim would return to the US to do some work, but he never lived there again and had no desire to do so.  And that’s when the story gets even more muddled. After he returned to the UK, he fathered Greg and, later, visited London for another job interview. When he returned home, he found his wife, Eliza, in a compromising situation and ordered her to sort herself out (I’m omitting some details). She did so by leaving Lynne and Greg with a neighbor and moving away. The children were then put up for adoption.    By one version of the story, he tried to claim his children but couldn’t since he was an American and not married to Eliza. But when I asked him about he said he gave them up because he didn’t have a lifestyle conducive to having children and he had no idea how to be a father. However, he was married again someone named Janath and, according to Lynne ...  [Jim probably gave Lynne and Greg up for adoption because] Janath instigated it, he paid maintenence to the local authority for our upkeep until we were adopted and sent presents at Christmas time.  Now, if you think the story was muddled before, it gets even weirder from here on out.  The Soviet Years  So Jim was with Eliza and our brother Greg was born early in 1969, but from 1970 to 1974, Jim was living in Moscow and met and married someone named Janath. Jim worked for a UK company, ICL (International Computers Limited)  , as a computer engineer, but behind the Iron Curtain  . This led to quite a bit of familial speculation about him. He was reluctant to discuss this and when he once sent me his “resumé for his life” (which I’ve sadly lost), he was fairly detailed about many of his jobs outside the Iron Curtain, but was quite fuzzy on what he did while he worked in the Soviet Union and Czechoslovakia. I do know that in the US Navy, he held a top-secret clearance as a Polaris Missile technician. Given that he was working for a British company behind the Iron Curtain, that would certainly upset the US government.  By Jim’s account, he was persona non grata at the US embassy while he lived in Moscow. Despite being fluent in Russian, he couldn’t exactly hang out with Russians because the Soviet government frowned on fraternizing with Americans. Thus, Jim’s “watering hole” was a bar at the British embassy because there was no one else with whom he could socialize. Here are a few tidbits that I remember from my conversations with Jim (paraphrased from memory, of course):  I was in the British embassy, having a drink, when I asked the waitress her name. She said it was “Janeth”. I was surprised because that’s such an unusual name and I told her that I knew a woman named “Janath”. I thought she’d find that interesting but instead, she got very upset. Her name was “Janice” and she though I was making fun of her lisp.  Jim would also tell a story of being at the British Embassy in Moscow and George Best  , was there. Best is considered to be one of the finest football (soccer) players in history and was also a raging alcoholic with a huge fan following in Moscow. He and Jim got drunk together, with Jim allegedly being the last man standing.   Jim had many talents and drinking was one of them.    And then there was the late night “drunken” email, that I asked him about later. He sent it to Lewis, Greg, and myself—but not to Gayle or Lynne—in 2010. I assume he was drinking at the time (he usually was) because he rarely shared personal anecdotes like this:  This movie - “Quiet days in Clichy” was cult when I was in Moscow in 1970 to 1974. All of the embassies had a copy and played it regularly. I got the tape from my friends Tony Conyers  of the Daily Telegraph and Dennis Blewitt   from the Daily Mail. It really brought back some fond memories when I found this on YouTube.  It is from a book by Henry Miller. “Clichy” is a suburb of Paris.  The link was to a preview of “Quiet days in Clichy” and, from what I understand, even though they played it at the British embassy, it was banned in the UK. I was very surprised by the email because our father rarely shared anything personal. He would sometimes share a joke, but this was part of his life. So, since I work extensively with the Web, I tracked down the film and create a private link he could download it from. And I watched it.  It was porn. I was happy that my father was reaching out, unprompted, to share something about himself. And it was porn.    A safe for work trailer for the movie “Quiet Days in Clichy.“.   After I sent him an email with the download link he replied “Thanks Curtis, You can take it down.”  And that was it. He sent the email at 11PM his time, apparently drunk, trying to reach out.  I’ve no idea what to make of this, other than knowing that he had feelings, but didn’t know how to express them. I asked him later and his “fond memories” were watching this banned film, with friends, in the British embassy in Moscow. From what we can gather, Moscow was a fantastic time in Jim’s life, full of screwing around, partying, and generally being Jim. It was a time of his life he seemed wistful for, though he did seem to suggest that he only had the British embassy bar as an outlet.  You have to keep in mind that this was during the Cold War between the USSR and the USA. Given that Jim worked on Polaris missiles and was now living in Moscow, speaking fluent Russian, the US government was not happy. Jim said it was laughable the few times he was approached by someone for casual chitchat and it would be obvious that US intelligence was again trying to find out if Jim was a traitor. Jim swears he wasn’t a spy and his being there wasn’t political. It was just a job.  Valerie  Lewis’ mom, Valerie, is an amazing woman and I’m happy to have met her. She was kind enough to share more details of Jim’s life.  A few years later, in 1975, Jim, still working for ICL, was living in Prague, Czechoslovakia, when he met Valerie, the mother of Lewis Poe. As Valerie tells it (and shared with permission):  I first met Jim in the British Embassy club bar in Prague and thereafter a couple of times in the Prague office of ICL for which UK company we both worked. We then ‘got it together’ and he arranged to work there on a more permanent basis so we could be together. After a few months we transferred back to the UK where he was sent on a training course and I worked in the centre of London. We lived together in north London and bought a flat in Harpenden before we moved to Germany. Lewis was born about 3 years later.  I was followed and stopped in Germany when I was with a friend as at that point my car still had Czech number plates. The Bader Meinhof gang who were active then were trained in Czechoslovakia, so I guess they were suspicious. A week or so later Jim and I were followed in Rudesheim by someone who made it obvious enough it was a warning of some kind. But I can assure you we were not involved in anything secret, underhand or exciting! We just fell into the category of suspicion as we had both worked in Eastern Europe and my politics were quite left of centre at that time. Cloak and dagger might have been fun!  She also writes:  One story which is possibly funny. Lewis was about 18 months old and we had been to see close friends in Holland who supplied Jim with a small amount of hash. Just before the German border (which we normally drove through without any check) we decided as an extra precaution to put the hash between 2 diapers which Lewis then wore. We got stopped and searched very thoroughly and just as I thought we were done for Lewis woke up and started to scream. Good old Lewis did the trick. They soon told us to go! I laughed all the way home.  So at this point, it’s 1979 and Jim is living with Valerie in West Germany. Jim sometimes had to work in East Berlin and he described passing through the infamous Checkpoint Charlie  , the only gate between West and East Berlin that foreigners were allowed to use.    Checkpoint Charlie, 1963. Source    As an American traveling to East Berlin on business, it was natural that he was going to be questioned. He told of handing his passport over and explaining Ich spreche kein deutsch (“I don’t speak German”), but apparently he said it so well that the East German officer didn’t believe him. The officer started getting upset and raising his voice with Jim. So Jim turned to the Soviet officer behind the East German officer and said in fluent Russian, “Would you kindly explain to this man that I am not kidding and I don’t speak German?”  The Soviet officer said nothing, but he smiled. The East German, however, upon hearing Jim speak Russian, immediately stood back. Jim said his passport came flying back to him like a frisbee and he was waived through, no questions asked. After that time, whenever it was that guard, Jim would smile and wave and the guard let him pass, never stopping him again.  And then the story takes a darker turn, as it often seemed to do with Jim.  When Lewis was around 3, Brenda became Jim’s secretary and he started his affair with her. He made life untenable for us telling me one day on the phone to get out as he would not be responsible for his actions if he found Lewis and I there when he got home. I have to say that I did love him very much when we were together. We had so many good times and he was truly happy about Lewis and used to tell me how blessed he was to have us both. When he met Brenda he turned from a loving husband and father to a bit of a monster. Sorry this probably isn’t what you need or want to know but unfortunately it was very much part of him.  Brenda was much younger than Jim and, I think, a little over ten years older than myself. Remarkably, Jim stayed with Brenda from 1981 until she passed away in 2006 from cancer.  Brenda  This is to honor Jim’s life, but doing so without saying a bit about Brenda would be to dishonor Jim’s memory, as she was the final love of his life. They planned to retire together in Greece, but she tragically passed away from brain cancer in 2006.  When I was living in Amsterdam in 2001, I took a train to Germany to meet Jim for the first time. We talked on the phone first, and he explained “Brenda isn’t much older than you and she’s very beautiful. When you’re here, I want you to think of her as your mother .”  This was Jim, being Jim. He was apparently so used to men chasing after any woman they wanted that he was concerned I’d make a move on his partner! I wasn’t happy he said this, but I wanted to meet the man.  When I arrived at their home in Idstein, it was a lovely, four-story home. And on one wall was a beautiful Keith Haring  artwork. I walked up to examine it and realized, with shock, that it was a painting, not a print.  Keith Haring paintings can sell for millions of dollars. Here was my father and Brenda, living in a lovely four-story home, with million-dollar paintings on their walls.  Except it wasn’t. Brenda was a forger, but a legal one. When she saw art she liked, she would simply paint a copy. It turns out that she previously worked for the Bank of England and her team would investigate forged banknotes. If the notes were good and the forger was caught, Brenda’s team would interview the person to find out where they obtained their ink, their paper, how they made the notes, and so on. Then they would replicate the notes and try to pass them to banks—always with a “get out of jail free” card in their pocket if they were caught.  For ten years after she left, whenever there was a major forgery in the UK, she’d get a visit from British police, verifying that she was not involved.    Jim also wrote the following about her work on Lotus Notes:  Brenda is doing fine. She’s as bogged down with work as I am. Her Lotus Notes Domino system is taking Europe by storm. She’s sold it to a sister division here - Abbott International “AI” (they opened a tender and then blind testing on 3 selected vendors. Brenda’s system won on both performance and price). They are the pharmaceutical side of Abbott. It’s revolutionised the German Affiliate where she works. IBM, who own Lotus, are taking a big interest and have committed to support if anything should happen to Brenda’s developer and his company. They are really doing some pioneering work with Domino and have a direct line to IBM support for any issues which come up. The developer has been made an IBM approved 3 party provider with guarantees backed by IBM. He’s laughing all the way to the bank.  She was an amazing woman. Sadly, when she passed, she wasn’t married to Jim,   so under German law, Jim had no rights to their shared assets unless he could prove what he paid for. A family member of Brenda’s claimed everything and Jim was left almost penniless. It was so bad at one point that Lewis was being sued for thirty thousand euro in “overpaid” child support because many payments were made from Brenda’s account. Fortunately, that was apparently dropped, but not before the lawyer suing Lewis tried to get Lewis to pay the legal fees.  Jim later was hired as a well-paid contractor for Abbott Pharmaceutical, alleviating his money woes for a while, but he never recovered financially, or emotionally, from Brenda’s death.  Brenda was indeed a lovely woman. And for all of Jim’s brusqueness, it was offset by her grace.  The Children    Your author and his sister, Gayle   Given all of that and Jim’s penchant for walking away from relationships or contacting his children, you might wonder how we all came to know one another. As with all things “Jim”, it’s complicated.  In 1987, I was living in Washington state, in the US, and had an odd hobby, skip tracing  . I did it for fun, helping friends find lost buddies and the like, but I realized one day that I might be able to find my father. It turned out to be remarkably easy and a few hours later, I was on the phone with a heretofore unknown cousin. Her father was Ron Poe, Jim’s brother. My cousin was telling me that Jim would send Christmas cards every year and she was kind enough to read off Jim’s address in Idstein, West Germany. Then Ron entered the room, asking his daughter who she was speaking to. He then took the phone and we chatted briefly. Ron wasn’t aware that I now had Jim’s address and told me that he hadn’t heard from Jim in years.  Though annoyed at the time, I later realized that Ron had no idea I was who I claimed to be on the phone. It’s only natural that he should protect his older brother. I’ve always wondered what he thought he might be protecting his brother from, though I have my suspicions.  Armed with an address, I was delighted to discover that West Germany also had an “information” telephone number to call to get the phone number for a given person and address. It was just after 5 PM local time, so when I called Jim, I managed to reach him just after 2 AM. Brenda answered the phone and after I explained, she handed the phone to Jim. I had finally found my father. His very first words are forever etched in my mind.  “How did you find me?”  He didn’t seem happy to hear from me, but I wrote it off to the shock of a long-lost child waking him up at 2AM.  After I got off the phone, I called my sister, Gayle, to let her know that I found Jim. I gave her his address and phone number and made her promise to wait until morning in Germany before calling him. Gayle promised. After we hung up, she promptly broke that promise, calling Jim immediately. It was not a good night for him.  Over the next few years I would call a few times, but Jim never seemed anxious to talk. It wasn’t until 1996 that I thought to ask an obvious question: do Gayle and I have any other brothers and sisters? Jim seemed very uncomfortable with this and finally said “David.”  At this point you might be wondering if you missed something because I’ve not mentioned David once, and for good reason. Lewis’ first name is David, but no one calls him that. Jim knew it because Lewis and Valerie lived with Jim and Lewis had gone to Germany and stayed with him on a holiday more than once. So me searching for “David Poe” wasn’t likely to turn anything up.  When I asked for more information, Jim told me about Valerie and where she might live in the UK: Southampton or Poole, on the southern coast. He claimed he didn’t have an address or phone number. This, I later learned, was a lie. Further, Jim did not mention Lynne or Greg.  I spent the next three years trying to track down Valerie and “David” and eventually found an address for Valerie. Unfortunately, her phone number was ex-directory (unlisted). Since I knew nothing about her, I was afraid that she might throw away any letter I sent, so I kept trying to find her phone number. And one day, in 1999, I was experimenting with software named Alphaworld  (one of the first online 3D worlds) and mentioned my tale of woe to another Alphaworld citizen. He was from the UK and told me to stay online. Half an hour later, he returned with the phone number of Valerie’s neighbor.      Lewis Poe   Very excited, I called the number and immediately ran into a new stumbling block. The woman I spoke with knew Valerie, but her son was Lewis, not David. Further, the woman insisted that no, Lewis didn’t have any brothers or sisters. I had to explain a couple of times that we had the same father, but different mothers and Lewis probably didn’t even know about me (it turns out he did but couldn’t find me). Eventually she said she’d go ask and a short while later, I was speaking to Lewis on the phone.  I later obtained a passport and flew to the UK, meeting him for the first time. We got along fabulously. And it turns out that Valerie’s awesome and there wouldn’t have been a problem with me sending that letter.    Greg Baddely   While visiting them, they informed me that there were two more siblings, but they’d been given up for adoption. They didn’t even know the genders. So I spent a few years trying to track them down. British adoption laws are strict, so there was no way I could get the information that way. The aforementioned Freedom of Information Act request for Jim’s records was, in part, to find any clue of who else he had known. But I was at a dead end. I finally gave up.  And then in January 28, 2005, Jim forwarded an email to Gayle and myself (which I forwarded to Lewis):  Jim - Hello from the past. I came across this person looking for you while surfing the net(at least I think it’s you she’s looking for). I keep in touch with [REDACTED] so had your e-mail address. Thought I would forward this to you rather than give her your address. Hope I didn’t F.U. How have you been? Would like to hear from you. Don’t know if I ever thanked you for pulling me out of the water at Lake Marion as I was going down for the third time. - Wes    Lynne Moore   Included in that email, was Lynne’s name and email address. When she got home from work that night, she had email from Gayle, Lewis and myself, brothers and a sister she never knew she had. She said she broke down and started crying.  Though it was kind of Jim to share that with us, Jim was still Jim. In his first email to Lynne he asked “How do you feel your life will be enhanced by knowing me?”  By a strange coincidence, I was being recruited by several British companies to help fix their IT systems. About a year and a half later, I had moved to Nottingham, UK, about an hour’s drive from Stoke-on-Trent where Lynne and Greg lived. Later, I moved to London and Greg and I shared a house, and later a flat together. I live in France now, but all the surviving brothers and sisters get on quite well. It’s the best part of Jim’s legacy.  Jim wasn’t all bad. When Lynne needed her car repaired, Jim bought her a new one. When Leïla and I got married, he contributed nicely to the wedding. He never understood what it took to be a father, so when he had money, he spent it on us from time to time, perhaps to make up for his past.  I also know he left behind many friends in Idstein, Germany. They’re great people and many of them are heartbroken he’s passed. Jim was never a good father, and clearly he was often not a nice man. But he’s all I had for a father, even if I didn’t know him well. Jim, wherever you are, I hope you’ve found peace.    Jim Poe, Curtis Poe, and Jim’s grand-daughter, Lilly-Rose.   Credits  I would like to thank my sisters, Gayle Poe and Lynne Moore, my brothers Greg Baddely and Lewis Poe, Lewis’s mom Valerie Poe, and my cousin Naomi Clements for helping to fill in some details. Thanks to my wife, Leïla, for proofreading and suggestions. There are many stories left out, some by request and others because they would cast living people in a bad light. Any errors and omissions, of course, are mine.  Update 1  Amongst other things, it appears that Jim lied about his birthdate. We the date as April 22nd, 1939. He was actually born April 22nd, 1941. There had been some earlier rumors that Jim might have lied about his age in order to join the Navy. If he was “one step ahead of the boys in blue”, as he put it, he might have had a compelling reason to lie, a lie he maintained for the rest of his life. Now that I’ve seen a birth certificate, albeit one issued in 1957, it confirms that he lied. It also raises some questions about his age when he was first married.  We’ve also found ourselves in the most delightful Catch-22. Germany won’t release a death certificate without a certified (apostille) birth certificate. Texas won’t issue that birth certificate because, now that they have been told he’s dead, the certificate can no longer be released unless it’s stamped “deceased,” for which we apparently need a death certificate.   Update 2  We’ve resolved the sitation with Jim’s death certificate (the US Consulate in Germany was very efficient in helping us), and Jim’s ashes are now buried and a tree should grow from it in the years to come. And in a parting gift, it turns out we may have one or two more siblings in Russia. . Not, I should say, a surprise.  ","title":"A Eulogy for My Father","url":"\/blog\/my-father.html"},{"body":"    Introduction  Astrobiology  Phosphine on Venus  Implications     Introduction  By now, if you’re a science geek like me, you’ve probably heard the news that Venus shows possible signs of life  . Yes, hellish Venus, of all places, might harbor life. This is due to the detection of significant amounts of PH 3 (phosphine) in the atmosphere. While the original paper downplayed the possibility that possibility that this phosphine was generated by life, nonetheless, it’s worth understanding the excitement and what it could mean.    The planet Venus.   Astrobiology  For many, Venus doesn’t seem like a likely candidate for life to exist. The surface temperature is 450&deg; Celsius (842&deg;Fahrenheit—hotter than Mercury!). The atmospheric pressure at the surface is 90 to 95 times greater than on Earth. What’s more, on the surface the atmosphere is overwhelmingly carbon dioxide and nitrogen. But you wouldn’t suffocate. You’d burn and be crushed to death first.  Despite that, the upper atmosphere of Venus is the most Earthlike in the solar system, with even NASA discussing ideas for exploration or colonization  .     So what was so interesting about the recent research paper?  In biology, there’s a subfield called astrobiology. I like to think of astrobiologists as science fiction writers with advanced degrees in biology. They consider the possibility of life elsewhere in the universe, what it might be like, and how we can detect it. One of the things they frequently discuss is biosignatures  . These are chemical signatures which, if they exist, suggest some form of unusual chemical process that might indicate life. For example, large amounts of free O 2 (oxygen) in the atmosphere would be a huge biosignature, as oxygen reacts with just about everything and is quickly removed from the atmosphere. While there are physical processes which can cause oxygen to remain free in the atmosphere, we have so much free oxygen in our atmosphere because life produces it. The detection of large amounts of it in an alient atmosphere would certainly be a great indicator that further investigation is warranted.  Methane is another biosignature gas because its existence is usually associated with life. We’ve been discovering lots of methane on Mars  and we don’t know why. Given that the 1976 Viking landers possibly detected metabolism on Mars  , the strange discovery of methane on Mars merits closer investigation. We don’t know what’s going on with Mars and life, despite it being very unlikely, could easily explain what we’re observing.    The planet Mars.   However, it cannot be stressed enough:  If the question is “aliens?”, the answer is almost certainly “no.”  Seriously, stop thinking it’s extraterrestrial life. Every time we’ve gotten our hopes up, they’ve been dashed. And that’s a good thing: it means science is working; we’re not searching for fun, we’re searching for truth.  Phosphine on Venus  Now, about that paper  .  The paper is fairly easy to follow, even if it’s aimed at a technical audience.  The authors of the paper were aware that phosphine is known as a powerful biosignature  . While phosphine can be produced in energetic reducing atmospheres such as Jupiter and Saturn (and has been detected there), rocky planets such as Earth and Venus shouldn’t have significant phosphine in their atmosphere, though Earth does due to phosphine production by extremophiles. However, we can potentially detect phosphine on exoplanets, so the researchers decided to use Venus to create a benchmark for further studies, but they found ”unexpectedly, our initial observations suggested a detectable amount of Venusian PH3 was present.”  This was a surprise. So the paper focuses on two areas. First, they considered all the possible ways that whatever they detected was not phosphine. After eliminating all sources of confusion they could think of, they concluded that, yup, it was probably phosphine.  The next part of their paper asks the big question: what could cause this?  They considered many, many different ways in which PH 3 could be created, but there simply isn’t enough free energy on Venus for that to happen in any significant quantities. For what little PH 3 could exist, they considered the lifetime of the molecule and outgassing, again finding an unsustainable amount of PH 3 . So where does it come from?  Here’s an important section of their research.   Even if confirmed, we emphasize that the detection of PH 3 is not robust evidence for life, only for anomalous and unexplained chemistry. There are substantial conceptual problems for the idea of life in Venus’s clouds—the environment is extremely dehydrating as well as hyperacidic. However, we have ruled out many chemical routes to PH 3 , with the most likely ones falling short by four to eight orders of magnitude To further discriminate between unknown photochemical and\/or geological processes as the source of Venusian PH 3 , or to determine whether there is life in the clouds of Venus, substantial modelling and experimentation will be important. Ultimately, a solution could come from revisiting Venus for in situ measurements or aerosol return.   In other words:   The PH 3 (phosphine) is not strong evidence of life  There could be previously unknown chemical processes producing it  It would be difficult for life as we know it to survive in Venus’ upper atmosphere  We know of no natural (non-living) processes which can produce this phosphine on Venus   There’s actually been plenty of prior discussion about the possibility of life in Venus’ clouds, with Carl Sagan writing about this in the 1960s. Serious research articles on the topic appear regularly  and pop-sci articles love to describe scientists pointing out that dark patches in Venusian clouds could be micro-organisms  .  Implications  While it’s very unlikely, what does it mean if life is found on Venus? That would depend on its origin. If we found that Venusian life had the same DNA or RNA as Earth life, that would strongly suggest a common origin. The panspermia  hypothesis, long laughed at in science, is looking more and more credible all the time, particularly since we find Martian meteorites on Earth. Wouldn’t it be fascinating if life originated on Mars and made its way to both Earth and Venus?  While I love that thought, there’s the more exciting possibility that potential Venusian life shares little, if any, biological similarity to Earth life. If so, that would suggest a separate origin. The implications of that would be staggering.  Currently, we only know of one place in the entire universe where life was created: Earth. If we had two independent creation events in the same solar sytem, this would imply that it’s common enough that the universe should be teeming with life. Absolutely teeming. As our technology improves, it would be easier to detect those biosignatures on the thousands of alien planets we’ve discovered. It would be amazing.    The Pioneer Plaque Source    As for intelligent life elsewhere, that’s a different story, but if the universe is teeming with life, that tremendously raises the chance for intelligent life elsewhere.  I can imagine few scientifics endeavors more important than sending new missions to Venus and Mars to investigate one of the most fundamental questions humanity faces: are we alone?  Update : new research indicates that the alleged phosphine detected in Venus’s atmospher is probably sulfur dioxide. Here’s the paper  and here’s a news article about it  .  ","title":"Life on Venus?","url":"\/blog\/life-on-venus.html"},{"body":"    Summary  History  Marx  Words Mean Things  The Nordic Model  Types of Socialism  Utopian Socliasts  Ricardian Socialism  Marxist Socialism  Democratic Socialism  Social Democracy  Social Market Economy  Possibilities     Disclaimer : I do not believe in Marxist socialism; I don’t have a problem with a social safety net, however.  Recently on Facebook—words which often do not bode well—a friend of a friend was not happy that his personal definition of socialism wasn’t accepted as a universal definition:   The government owns and controls the means of production; personal property is sometimes allowed, but only in the form of consumer goods.   Because his personal definition wasn’t accepted, he couldn’t create strawmen arguments to knock down, nor could he meaningfully engage in debate because, gosh , it was simply unacceptable that the definition he just Googled wasn’t good enough for the other debate participants.    The Old Port of La Rochelle Source    Except that this definition, for what it’s worth, is the one that is largely accepted today in the US. Meanwhile, in La Rochelle, a lovely medieval town on the West coast of France (one in which I used to live), they’re expanding the harbor to provide better services to billionaire’s super-yachts. It’s socialists who voted to do so because La Rochelle is undeniably a socialist town. Turns out the socialists here in France often have no issue with private property or money, but they strongly object to the abuse of either.  For Americans, it gets even more confusing when self-described “democratic socialist” Bernie Sanders says “thank God” for capitalism.     So what gives? Do socialists agree that private property is OK or that it should be abolished? Can’t have it both ways, can you?  Summary  Because, sadly, modern people’s attention spans for reading are short, I’ll share the definition of socialism now and you can decide if you’ll continue reading or not.   Socialism: a political or economic system that promotes social welfare.   That’s it. And it’s not exclusionary. If we look at the social democracies known as the Nordic Model, they are very pro-capitalist and promote business welfare, too.  So how do we get from government controlling the means of production to this definition? That takes a bit of explaining.  History  You can’t have a meaningful discussion of the meaning of socialism unless you understand the origin of the idea. And no, Karl Marx didn’t come up with it. His most famous work, Das Kapital  , was published in 1867, but it was fifty years earlier, possibly with the publications of work by Robert Owen  , that socialism was gaining traction. However, you can’t even appreciate this without context.    Adam Smith Source    In 1776, Adam Smith published The Wealth of Nations  , a book today regarded as one of the first books describing modern economics. And honestly, it had to be. The foundations of modern economics are land, labor, and capital. Historically, land wasn’t freely bought and sold. It was largely held by nobility who would no more think of selling their ancestral properties than you would think of selling your parents. The concept of laborers freely seeking the best possible wage also didn’t exist. In the middle ages, most people were serfs, tradesmen, clergy, or nobility. Fiefs, lands to which serfs were tied, still existed in the 17th century  . And utilizing capital to leverage your wealth to build even more wealth wasn’t really a thing because modern economic thought was, well, not thought of. In fact, the innovative possibilities of capital combined with profit-seeking labor is very much a modern invention.  Adam Smith came along when economies were transitioning from zero-sum Mercantalism to industrialized capitalism. With the birth of the industrial revolution in the mid-1700s, we started to see a magnificent growth in profits. We also saw a magnificent growth in new forms of human misery. In the paper entitled Time and Work in 18th-Century London  , Hans-Joachim Voth cites Herman Freudenberger and Gaylord Cummins as arguing that   annual labor input possibly increased from less than 3,000 to more than 4,000 hours per adult male between 1750 and 1800.   That’s up to twice the hours most Americans work today (and Americans tend to work far more hours than most other developed nations).    Child Labor Source    And frequently those laborers were children. It appears that many children were effectively slaves during the industrial revolution  . And by “slaves”, I mean that with all of the horrifying implications. Robert Heilbroner, in his famous economic history The Worldly Philosophers  wrote:   Indeed, a recital of the conditions that prevailed in those early days of factory labor is so horrendous that it makes a modern reader’s hair stand on end. In 1828, The Lion, a radical magazine of the times, published the incredible history of Robert Blincoe, one of eighty pauper-children sent off to a factory at Lowdham. The boys and girls—they were all about ten years old—were whipped day and night, not only for the slightest fault, but to stimulate their flagging industry. And compared with a factory at Litton where Blincoe was subsequently transferred, conditions at Lowdham were rather humane. At Litton the children scrambled with the pigs for the slops in a trough; they were kicked and punched and sexually abused.   Sadly, these accounts of the horrors of child labor during the industrial revolution are not scarce. Is it any wonder that many people objected to this new idea of capitalism?  One of the most prominent objectors was the aforementioned Robert Owen. A highly successful businessman, Owen was aghast at how workers were treated and decided to buck collective wisdom and create a more humane business. In 1799, he purchased New Lanark Mills  and   Under Robert Owen’s management from 1800 to 1825, the cotton mills and village of New Lanark became a model community, in which the drive towards progress and prosperity through new technology of the Industrial Revolution was tempered by a caring and humane regime. This gained New Lanark an international reputation for the social and educational reforms Owen implemented. New Lanark had the first Infant School in the world, a creche for working mothers, free medical care, and a comprehensive education system for children, including evening classes for adults. Children under 10 were not allowed to work in the Mill.   And Heilbroner reports that from 1815 to 1825, there were over 20,000 visitors, including nobility and business owners, all trying to figure out how this mill had become hugely profitable.  Owen, along with Henri de Saint-Simon  , Charles Fourier  , and others, formed what we today call the “Utopian Socialists.” Generally speaking, they were a reaction to the brutality of the industrial revolution and these thinkers envisioned better worlds, or utopias, based on decency, kindness, and social needs. However, with the possible exception of Owen’s direct action in New Lanark, these were imagined worlds, built on dreams, not evidence.  They were, however, the first socialists, they were all over the map, and they had little to do with governments controlling the means of production.  In contrast to them were the Ricardian Socialists  . They focused quite heavily on the labor theory of value  which argues that the value of a good or service is the total amount of labor required to produce it. Yes, many people think that’s Marxist because Marx was quite fond of it, but their beloved Adam Smith championed it! In Chapter V of Wealth of Nations  , he wrote:   The value of any commodity, therefore, to the person who possesses it, and who means not to use or consume it himself, but to exchange it for other commodities, is equal to the quantity of labour which it enables him to purchase or command. Labour therefore, is the real measure of the exchangeable value of all commodities.   The Ricardian’s (so-called because David Ricardo  also accepted the labor theory of value and his arguments heavily influenced them) argued that because value is derived from labor, those who labor deserve the rewards. This is curious to me because it often implies that managers and owners who supervise and direct physical labor—often without doing it themselves—are somehow less valuable, but I digress.  To the Ricardians, labor should reap the rewards of commerce, but is instead exploited. One of these was Thomas Hodgskin  and the opening paragraph from Wikipedia sums up many issues quite nicely (emphasis mine):   Thomas Hodgskin (12 December 1787 – 21 August 1869) was an English socialist writer on political economy, critic of capitalism and defender of free trade and early trade unions. In the late 19th and early 20th centuries, the term socialist included any opponent of capitalism , at the time defined as a construed political system built on privileges for the owners of capital.   For Hodgskin, capitalism was built and maintained to serve the privileged class. However, he didn’t object to capitalism per se . He objected to what we might call crony capitalism  . He didn’t want a world where one prevailed due to government meddling and backdoor deals; he was, in fact, a Libertarian who loved the market system, but championed labor. He insisted on the rights of unions to exist and advocate for workers (because which Libertarian would object to free association?) and he strongly objected to rent-seeking behavior  . But he was definitely a capitalist, through and through. And a socialist in that he advocated for the needs of society, not just the monied class.  Marx    Karl Marx Source    It wasn’t until Karl Marx came along and skewered the Utopian and Ricardian socialists, that people started thinking of socialism in terms of Marx’s more specific view. This was, in part, because in The Communist Manifesto and other writings, Marx and Engels created a more detailed description of, and argument for, socialism. They both acknowledged the moral considerations of the Utopian and Ricardian socialists but destroyed them for being fantasies.   And given that the Manifesto called for revolution and swayed many people, Marx’s views on socialism were indelibly etched in people’s minds.  In particular, in The Communist Manifesto  Marx wrote of the Utopian socialists:   By degrees, they sink into the category of the reactionary [or] conservative Socialists depicted above, differing from these only by more systematic pedantry, and by their fanatical and superstitious belief in the miraculous effects of their social science.   But what of his own views? In the Manifesto, Communism is the result, but socialism is a necessary step:   We have seen above, that the first step in the revolution by the working class is to raise the proletariat to the position of ruling class to win the battle of democracy. The proletariat will use its political supremacy to wrest, by degree, all capital from the bourgeoisie, to centralise all instruments of production in the hands of the State, i.e., of the proletariat organised as the ruling class; and to increase the total productive forces as rapidly as possible.   And that is effectively the definition that many hurl against socialism today: naïve workers gutting honest, hard-working, private enterprise in favor of an inefficient nanny state. Or to phrase it in the other extreme: the proletariat rising up against the bourgeoisie to put power in the hands of the people.  Personally, I don’t buy either definition because a) I don’t see it happening and b) I can’t help but roll my eyes at those those think the definition of an ever-evolving word must be set in stone by an essay written 172 years ago, ignoring all modern thought on the subject.  Words Mean Things  Yes, words mean things and then they mean other things. For example  :   Girl meant any young person – “gay girl” meant girl, not lesbian, and “knave girl” meant boy – while “man” meant any person, regardless of gender.   The word “girl” was originally gender neutral and arrived in the English language around 1300 CE  , but we’re not certain of its origin.  Or consider that the word “meat”, coming from the Old English maet , simply meant “food”  and its current definition of “animal flesh” didn’t arrive until the 1300s.  Words, of course, have no intrinsic meaning. When I hear the word “peer”, I think something like “equal in quality”, but if I’m speaking French, the word « pire » has the same sound as “peer” but means “worst,” a radically different meaning.    Words have the meaning we ascribe to them and nothing more. But this causes many problems if we don’t understand that another’s definition is different from our own. For example, some people think “meat” only means pork or beef, leading to ridiculous incidents of vegetarians being served chicken because they “don’t eat meat.”  So if you have a definition of socialism that doesn’t match my own and we don’t realize this, it’s very easy to have a confused argument. So what do people think socialism means today?  According to a 2018 Gallup poll of Americans  :   &nbsp; 2018 1949  &nbsp; Sep 4-12 Sep 3-8  Equality - equal standing for everybody, all equal in rights, equal in distribution 23% 12%  Government ownership or control, government ownership of utilities, everything controlled by the government, state control of business 17% 34%  Benefits and services - social services free, medicine for all 10% 2%  Modified communism, communism 6% 6%  Talking to people, being social, social media, getting along with people 6% --  Restriction of freedom - people told what to do 3% 1%  Liberal government - reform government, liberalism in politics 2% 1%  Cooperative plan 1% 1%  Derogatory opinions (non-specific) 6% 2%  Other 8% 7%  No opinion 23% 36%   It’s interesting to note that “government control” has now slipped to second place between “equality” and “benefits and services.”  And then we have this lovely New Yorker article discussing how tangled the definition of socialism is in America today  .  The Nordic Model  Or consider the Nordic Model  . The social democracies which have adopted the Nordic model have strongly embraced globalization, capitalism, and social welfare. While they differ in varying degrees, key similarities in all of these countries include:   a comprehensive welfare state  large-scale investment in human capital, including child care and education  labor market institutions that include strong labor unions and employer associations   Depending on where you want to draw the line, the countries which follow the Nordic model are Finland, Denmark, and Sweden, but many include Norway and Iceland in this group. These countries tend to have strong economies and, most importantly, five out of the seven happiest countries in the world  follow the Nordic model. I don’t believe that is a coincidence.  Given the historical definitions of socialism, I think the social democracy of the Nordic Model fits quite well.  Types of Socialism  So now I think we can break socialism down a bit into a few different types. The following, of course, is by no means exhaustive and in some cases, is a touch at odds with definitions provided by economists.  Utopian Socliasts  Largely a reaction to the brutal industrial revolution capitalism, the utopians envisioned perfect societies, all radically different from one another. They’re generally considered to be naïve.  Ricardian Socialism  Also reacting to industrial revolution capitalism, the Ricardians accepted David Ricardo’s description of the labor theory of value and argued that all wealth belongs to the laborers. They tended to reject ideas such as rent or profit, but some, such as Hodgskin, were also capitalists.  Marxist Socialism  This is what most Americans mean when they say “socialist”, which is curious because it’s a dying breed in the West (though still somewhat popular here in France). In short, Marxist socialism is when the government owns the means of production of goods and services. Marx also called for the abolition of private property, but not personal property (consumer goods).  Democratic Socialism  In this model, democracy is paramount, but it’s combined with a drive towards a socialism closer to the Marxist ideal. There are no countries which currently follow this model.  This is what Bernie Sanders claims to be, but in reality he fits the social democrat model better.  Social Democracy  I like to think of this as “modern socialism”. There are several strands, but a key feature of Social Democracy is embracing capitalism while trying to contain its worst excesses, possibly via nationalization of problematic industries. Strong social safety nets are a key component of this. However, this leads us to ...  Social Market Economy  Though not a form of socialism in the classic sense, I feel that it fits the zeitgeist of feelings on socialism today.  This is what much of Europe has adopted today, including the economic powerhouse, Germany. It’s similar to Social Democracy, but tends to shy away from nationalizing industries. Social Market Economies tend to support the market, but not guide it, and have very strong social safety nets. They also strive to constantly understand the economy and adjust and adapt, rather than try to find a one-size-fits-all economic “Theory of Everything.”  Possibilities  Imagine a world where you don’t go bankrupt because you tripped and broke your leg. Imagine a world where you don’t have to choose between forgoing college or decades of crippling student debt. Imagine a world where you’re paid a living wage as a waiter—or you can get filthy rich building your own business empire. That’s socialism today.  And an overly simplistic review of the meaning’s evolution over time:   Pro-social and sometimes pro-capitalist (Utopian and Ricardian socialists)  Government control of production (Marx)  Pro-social (today)   Yes, there are still people who will pedantically insist that Marx’s definition is the “one true socialism”, either because they agree with it or they wish to attack it. But his definition wasn’t the first, nor will it be the last. And while economists (and pedants) still cling to Marx, many of us are looking to the future and see the potential of caring about people as much as business.  ","title":"What Is Socialism?","url":"\/blog\/what-is-socialism.html"},{"body":"   This is more of me sharing tidbits of my life so that my family can know more about my past. I don’t want to be the enigma that is my father .   The year was 1978. I was eleven years old. My family was living in a run-down complex of duplexes in Schertz, Texas, and my only friends in the complex were the McClouds. They were a couple of boys, one my age and one slightly older. They were the Platonic ideal of “juvenile delinquent” and my mother didn’t like me playing with them, but there weren’t many children in the complex and she wasn’t home at the moment.  We were playing tag and I was running from the younger McCloud, desperate to not be tagged “it.” He was chasing after me because, as the least athletic, I was the natural target. I was skinny, but I was clever. I was running towards the ivy-covered chain-link fence in our back yard. I was betting that if I could jump that fence, he would chase an easier target. As I grabbed the top rail, I vaulted and soared over the fence perfectly. I also felt a wasp nest beneath the leaves crush under my hands.    These things scare the hell out of me.   Despite not seeing it, I knew instantly what it was. I can’t convey the visceral terror that I felt at that moment. I went from being the least athletic to the most as I ran for my life. The wasps, meanwhile, hastily convened a meeting to discuss how to redress such a gross violation of their sovereignty and decided to do what wasps do. I was right to run.  As one of them reached me and stung my arm, I fell to the ground, screaming as much from terror as from pain. And I knew with a dead certainty that the wasps would be swarming over me, stinging me again and again. What happened was far worse.  The younger McCloud had also vaulted the fence. He ran up and kicked me, screaming “you’re it” as he ran off. It was not a good day.  If memory serves (and being almost five decades ago, it most certainly does not), I was only stung once, but that, and feeling the wasp nest crush under my hand, was enough to instill a deep, lifelong terror of them.  Fast forward a few years and I was spending summers on Alvin Wittig’s farm. He was a close friend of my mother and Alvin and his father were happy to have me spend a few weeks there. I’d read decades-old copies of Reader’s Digest that his father kept on a bookshelf, sometimes marveling that an article written in the 1950s would be reprinted decades later, but with a new author’s name and the dates and times updated. I’d help Alvin herd cattle. I’d mess around in his tool shop. I’d ride a demon creature from Hell, a pony named Queeny, he had rescued from a family that treated it poorly. To this day, I’m sure science has equines wrong. They’re actually hivemind creatures, all of them networked together to communicate a shared desire to kill me. Queeny used to cheerfully do a “stop-drop-and-roll” in rocky fields if I tried to ride her. She’d buck and jump and rub against trees to get me off her. Alvin laughed.  Other equines had similar attitudes towards me. The gentlest nags in the world would take off at a gallop if I mounted them, but we finally reached a compromise: I agreed not to ride them and they agreed to stop trying to kill me. My daughter loves these demon creatures from hell. She takes horse-riding lessons while I watch, with a fixed smile on my face, awaiting the moment the horses realize she’s my daughter. Fortunately, the horses are too stupid to make the connection.  Where were we? Oh yes, wasps!  No, not yet.  Alvin was also a beekeeper. To me, bees were little short of thalidomide-baby wasps, runty bullies whose only saving grace is that they would die if they stung me. There were always plenty of bees flying around Alvin’s farm and I’d swat at them and Alvin would patiently explain to me that if I left them alone, they’d leave me alone. He’d be explaining this to me, with tenderness and affection, as he’d remove yet another bee stinger from my skin. Alvin, requiescet in pace , was a good man.  Alvin would show me how to smoke the hive, remove the honeycomb, cut off the top layer and spin it in a centrifuge to remove the honey. Then he’d cut off a piece of honeycomb for me to enjoy like bubblegum. I never liked honey, but I never told Alvin that. Eventually, I got the point where I’d rest my hand at the entrance to the hives, the bees crawling all over me while they made their way in and out. They never stung me again and I found a strange sort of peace, watching this erstwhile terror go about the business of living.  Fast forward 20 years later and I found myself a barista, working a beach event where I was managing other baristas serving overpriced coffee to tourists on an Oregon beach and when the bees started swarming, our customers ran away screaming, frantically batting at the bees. I instructed my staff to start pouring circles of vanilla syrup onto paper plates and setting them off to one side. The bees happily swarmed the plates, a bobbing collar of lace absorbing the sugar. They were beautiful. When the bees would swarm again, I’d call out, “time to feed the bees,” and they’d pour more syrup for them, keeping the tourists safe from nature.  But wasps. Fuck them.  Today, I live with my wife and daughter in a house in the southeast corner of France, about 45 minutes from the Italian border. I’m blessed with a wonderful family, a paradise to live in, and wasps. Did I say, “fuck wasps”? It bears repeating.  We live in the countryside. You might even say in a forest. We like to eat outside, but this is a siren call to the wasps. Last year, we’d frequently find ourselves running inside when the wasps muscled in, delighting in the smell of our barbecue and the meat that they crave. The bees would show up, too, but they didn’t bother me as much any more. But the wasps? Fuck wasps.  I hunted for their hives to no avail. We set up a string of wasp traps around our home, killing many of these foul, evil beasts, but not stopping the invasion. There was a war on and the dominant species on the planet was losing it.  The winter bought us some respite, but thanks to the never-ending wonder of climate change, France experienced an unusually light winter. When the leaves fell last winter, we found two huge wasp nests the size of basketballs, several stories above the ground in tall trees. I wasn’t worried because those nests were dead. I was wrong. The queens that should have died in the frost did not.  This year, 2022, is being called The Year of the Wasp  .  Due to the heatwave, we’ve been leaving the windows of our home open. The wasps are now coming in for cat food. And heaven help us if we try to cook anything because the battle ensues. On my best day, I killed nine wasps. Then I went out and refilled six wasp traps with the sweet fluid that leads them to their well-deserved deaths. The wasps traps are filled with dead wasps, but refilling them is a challenge because they’re often filled with live wasps. And more keep coming.  We’re losing this war. We need to a different approach.  We could just refuse to eat outside, but we’re not giving up our quality of life for damned wasps. Even if we did eat inside, we’d have to close all of the windows. Not an option in this heat. Call exterminators and ask them to scour the forest for wasp nests? I don’t think so.  My wife came up with a different approach. Why kill the wasps? They have a right to live, too. She decided to try a peace offering. We offer a bit of food for the wasps, away from where we’re eating. After they swarm the food, we sit down to eat in relative peace.  So far, it mostly works. Our hot war became a cold war and is becoming a détente. We still have wasps dive-bombing our plates from time to time, but we’re learning to be at peace with them. It’s not easy as they still scare the hell out of the eleven-year-old boy in me, but all of us are getting more used to them. We’re becoming a wasp sanctuary.  My instincts tell me to kill them or run. My conscience tells me we’re doing the right thing.    Wasps eating our peace offering.   Last night, we ate sushi. We put aside a little platter of raw fish for the wasps and left them alone. When they settled down to eat, we ate our meal in peace.  Be safe, little wasps.  ","title":"The Wasp Sanctuary","url":"\/blog\/the-wasp-sanctuary.html"},{"body":"    The Early Days  Barista  My First Test  How I Became a Bill Collector  Sexual Harassment  Leaving     The Early Days  Note: As usual, much of the below is from decades ago and my memory isn’t perfect. In the unlikely event you are mentioned here, it’s possible your memory will be different from mine.   I am often asked how I got into writing software for a living. This is the story of my first real programming job and the strange path that took me there.  In 1982 I was attending Keystone School  in San Antonio, Texas. It was a very dark time in my life and my geometry teacher recognized that I was in trouble, so she allowed me to play in the computer room that was reserved for older students. I learned BASIC programming and discovered something in school that I loved. After I left that school to go back to a public school, I started writing (bad) games in BASIC and laboriously typing in programs I found in magazines.  I had begun with Commodore PET computers and moved on to the technological marvel that was the TRS-80 (called the “trash 80”). By 1985, I already written a fully-featured lo-res graphics program for the TRS-80 that allowed users to create images, saved across multiple files (I didn’t understand data compression back then) and stitch those files together to print the images on a dot matrix printer.  Those were the days. I had demo images with that, including a nice, but pixelated minotaur’s head, dragon’s head, and other AD&amp;D images.  One would think that after I graduated high school in ‘85 that I would go on to college and do something with my life, but one would be wrong. We were dirt poor and at the time, to qualify for financial aid for college, I had to first apply for a Pell Grant.    That required that my parents provide several years of past tax returns to show what level of financial support they could provide. My mother explained to me that my step father did the taxes. He had failed to file them for years, so no Pell Grant for me. No financial aid for me. No college for me.  The next several years were me bouncing around from odd job to odd job, mostly fast food. At one point I had worked at Wendy’s  restaurant in Anchorage, Alaska, Kingsville, Texas, Universal City, Texas, Longview, Washington, and Honolulu, Hawaii. I was a pathetic peripatetic, wandering from state to state, deeply embarrassed that the “Most Intelligent of the Class of ‘85” of Marion High School was an itinerant burger flipper.  Years later, I eventually went to a community college in Portland, Oregon and managed to get an Associate of Science degree with a 4.0 GPA   , having taken every economics class they offered with a specialization in international relations.  An Associate’s Degree is a two-year degree and unless you use it to go on to a four-year college, it’s worthless for many jobs.  Barista  Which is how I found myself working as a barista (espresso bartender) when a young, pretty lady walked up and placed her order, leaving a large tip in my tip jar. I was proud of the fact that I was the highest tipped male barista in our chain of stores and I accomplished this by flirting. So I flirted with her, in my naïveté thinking it was innocent fun. I had known her and her boyfriend as regular customers, but that’s all. And that’s when she made me a rather curious offer.  I declined, pointing out I had a girlfriend, to which she replied that my girlfriend could join us.  Er, no. So when her boyfriend arrived later to place an order, I apologized, explaining that through I didn’t know him well, he should know what happened.  He swore, thanked me, and explained that their lifestyle was a bit different. However, they agreed not to proposition people without the other’s knowledge. I still wasn’t going there, but he and I started talking more and eventually we became friends. Turns out he was a software developer and when he found out I used to write software, we started talking about this frequently.  One day he came in to place an order and said he was quitting his job and he’d be happy to recommend me. The software was written in something called UniBasic  and he was sure I could handle it. He would stay on for three months to train me.  I got the job and it got my foot in the door, but to my surprise, it was often less professional than the fast food restaurants I had worked at.  My First Test  My friend was leaving his job on the last day of December. I had been on the job for just three months, learning their software, but only being allowed to work on it part-time. Otherwise, I made sure all of the employees were turning in their paperwork correctly. On that last day, the controller called me into his office. He explained that they needed to increase the price of everything by X%, varying per category, and have it ready for tomorrow.  This wasn’t just running a bunch of SQL. We were using a UniData multivalue database  , prices were scattered all over the place, and I wasn’t very good with their query language. Worse, our license allowed us to write code to read the data, but not change it. Updating this data was done through software running on a terminal and it created audit trails for everything. I needed audit trails, too.  This was a bad idea on many fronts. First, my friend made it clear how much he despised the place and he was leaving at 5PM because he had a New Year’s Eve party to attend. He explained to me how to do the job and I knew enough about the design of our database to figure it out, but it should not have been a rush job.  Next, since software I wrote could only read data, not change it, we had a huge risk. If we changed the data and were caught, we’d have to pay $25,000 (almost $50,000 in today’s money) to buy a copy of the software outright. We were a small company and that would have hurt us. We’d also lose all technical support from the company that “designed” this software.   and we’d no longer get free upgrades. Having a few hours heads-up on New Year’s Eve that I needed to risk destroying the company’s business wasn’t fun, but this was how the company worked. So I worked.  After writing the software to update the prices, I ran it. If it worked, it would look like people had spent days manually updating all of the prices, complete with audit logs. I had done it in hours. But if I—a junior developer—corrupted the data, I might be fired. Happy F’ing New Year.  It was getting late in the evening and I was tired and stressed. I ran the software. I corrupted the data.  The only reason I didn’t get fired is that I wrote the software to terminate after the first change and I had printed out all of the data that would be altered by the first run (this seeming much easier than the alternatives in those early days of software development). The controller and I manually fixed the data, I figured out what I did wrong, and ran my software again. This time it worked.  I went over all of the data carefully. I double-checked everything I could. The database was complex and prices for different kinds of goods were stored in different ways, but I couldn’t see anything wrong with what I did. I explained everything to the controller (who, to be fair, had become a friend of mine. He was also surprised by this last minute request). I told him I could remove the code that caused the software to halt after the first change. The software would run overnight and we wouldn’t know until the morning if it worked. The controller told me to do it and get to my party.  January 1st, I get a phone call early in the morning. I’m hung over and not looking forward to this call. It seems they had been checking all of the data they could.  The software worked.  How I Became a Bill Collector  This was a small, family-owned business, with several stores offering a variety of products, many of which had to be ordered months in advance to be handmade in Europe and shipped to the US. The business wasn’t struggling financially, but as far as I could tell, they weren’t getting rich. They needed more money. In fact, I needed more money. I asked for a raise.  At the time, I was paid $x dollars an hour for my administrative role and a couple of dollars an hour more for when I wrote software, capped at 15 hours a week. This wasn’t legal, but I wasn’t slinging espresso or burgers. I never complained, but I still asked for that raise.  We had a new office manager who was not well-liked, was a rubbish manager, but knew our products like the back of his hand, having recently been promoted from warehouse manager. He listened to me, nodded, and explained that they were having trouble with customers ordering goods and not paying, leaving the company with a huge accounts receivable (AR) balance. The company simply didn’t have the money to pay me.  Oh. That’s interesting. In my prior life, I had also worked in car sales. I knew exactly what this was: a negotiation. The manager probably thought he had told me “no,” but I heard his objection and I isolated it  .  Me: “You don’t think I deserve a raise?”  Him: “Yes, you do, but we can’t afford it.”  Me: “And this is all because of the outstanding accounts receivable?”  Him: “Yup, sorry about that.”  Me: “So if I can get those people to pay, you can afford to pay me.”  He was trapped by his own words, but to get out of that, he made me an offer I couldn’t refuse, but couldn’t win. If I could get AR to zero by a certain date, I’d get a raise. He wouldn’t budge from zero. We had a lot of outstanding accounts, but not an insurmountable number. However, the goods we sold weren’t cheap and getting people to pay wasn’t going to be easy. I was allowed to take any reasonable steps to enforce this.  That’s how I became a bill collector.  The first thing I did was, of course, write software to produce daily reports for me on the next accounts I would go after, and the steps I had taken to collect on those accounts.  I sent email to everyone I could, and called the others (all customers gave us phone numbers). An astonishingly large number of those people had no idea they owed money and paid up. No one had bothered to contact them! I was surprised at how easy it was. But a large number of them simply didn’t answer.  My next step: I received permission to contract with an actual bill collector. They would buy the debt from us, paying a fraction, and then it was their problem. That’s when I discovered the world of potential clients wining and dining you to win your business. I only accepted lunch from them, but now I realize that even that is dubious.  I chose a bill collector, the company signed the contract, and another large subset of accounts were cleared. But there will still more.  One, in particular, was someone who told me on the phone that she owed nothing. But she did, and I had the paper to prove it. In fact, I often went back to the paper because if this went further, I wanted more than just ones and zeros on a hard disk. She clearly owed the money and it was a lot, so I tried to collect directly rather than hand her off to the bill collector.  Turns out she was a lawyer and immediately replied that we were going to be sued for harassment. The company agreed to write her debt off and the grounds that a trial would cause more pain than her debt. Another account cleared.  On the day of my deadline, the manager greeted me with a smile. He was very happy with the work that I had done but, unfortunately, I hadn’t reduced AR to zero. No raise.  I handed him the AR report. If the owners of the company paid their bills for the products they bought, our AR would actually be positive. Should I send them to collections?  The trap was sprung, the manager was caught. I got my raise.  Sexual Harassment  The worst nightmare at this job, though, was the sexual harassment.  Three people, Alice, Bob, and Charlie. All of them were nice people and friends of mine. I was still very young and not understanding how business was supposed to work, so I flirted with Alice. She flirted back. We liked each other, but she was also an extremely devout Christian, so my atheism was an issue.  Nonetheless, we continued to flirt. A little harmless fun, right? Bob didn’t flirt with her, but he cracked lots of jokes, was well-liked, and he and I started hanging out after work. He was a great guy; we’ll get back to him. I liked Charlie, too, but Alice didn’t. So when Charlie would make inappropriate comments to her, just like I did , she ignored them.  One day she came to me with a problem. The company was driven by testosterone-fueled cronyism and she didn’t trust upper management. She said she was being sexually harassed by Charlie and didn’t know what to do.  Oh, shit. I wasn’t entirely sure of the law, but a few thoughts crossed my mind.   I stopped flirting instantly  If I didn’t act, I could be culpable  I had to talk to upper management  She was about to be fired for being our worst salesperson   She reluctantly agreed to let me explain the issue to management, but wouldn’t tell me details. After speaking with management, they were incredibly pissed that they couldn’t fire her. However, they would take no action against Charlie because he was our number one salesperson. They eventually agreed that Alice and I would have meetings to work the situation out, with me being the intermediary between her and upper management. I was not happy about this, but that was that. So some evenings, Alice would leave the sales floor and have private meetings with me in a locked back office, trying to find a way forward. This was not professional, but I didn’t know that. And remember, I left the sales floor and locked myself in an office with a pretty woman who wasn’t selling anything but still kept her job.  Bad optics. Really bad. Bob, my new friend who I was hanging out with a lot, joked in front of Alice and myself that he hoped we were having a good time in the back office. He knew nothing about the sexual harassment allegations and I couldn’t tell him. I just turned to him and replied, “never say that again.”  I lost a friend that day. It hurt.  It was Charlie’s flirting with her that she objected to. She found him repulsive. At this point, I knew Alice well enough that I pointed out that I had said similar things (a phenomenally stupid thing to say), but she explained that she knew I didn’t mean anything. Charlie, in her eyes, was a creep.  In reality, Charlie was gay and had no interest in Alice, but he still liked to joke around. Of course, that in no way excused his behavior any more than I had an excuse for my behavior. Alice didn’t know Charlie was gay, but as a devout, conservative Christian, it certainly wouldn’t have made the situation better.  Eventually, Alice left the company (I don’t know the circumstances), and I learned a hard lesson about my personal behavior and what is and is not appropriate in the work place. I also began to get an understanding of how horribly women are often treated and realized that, even if she liked me, I had a hand in that treatment. It’s not a fun lesson to learn about oneself, but the most painful lessons are often the best.  My flirting played a role in my landing that job, and now it played a huge role in several people being hurt.  Leaving  I started going back to college to take COBOL classes and after six months, leveraged that knowledge and my limited experience to get a full-time job at an insurance company. At the time, it was the best paying job in my life. I was a finally a professional software developer.  My first programming job is still my “best” worst job because it was the platform that launched my career, but it was a nightmare. Good people, bad people, and cronyism galore. And decades later, I still feel bad about my role in the mess.  ","title":"How I Landed the Worst Best Job","url":"\/blog\/how-i-landed-the-worst-best-job.html"},{"body":"    An unusual event for Blue Origin Source    With the publication of the open letter to Jeff Bezos from his employees  , Jeff Bezos is continuing to have a Terrible, Horrible, No Good, Very Bad Year. Despite the hopes of many in the space community to see rich competition amongst private space industries (since the government ones are moribund), Blue Origin’s completely inability to do, well, anything, has dashed those hopes. SpaceX is clearly the winner and all Elon Musk had to do was to uncharacteristically say nothing.  And Bezos' woes are self-inflicted.  Note: All of the following is grossly over-simplified.  I often talk about SpaceX and Blue Origin with friends and many are surprised to find out that Jeff Bezos founded Blue Origin in 2000  , two years before Musk created SpaceX. Bezos has been pouring a billion dollars a year into Blue Origin while SpaceX came close to bankruptcy several times in its early years. Blue Origin follows a traditional, slow process of incremental advances, while SpaceX is applying agile project management to rockets . Insane!  And SpaceX is destroying Blue Origin. Just this year, SpaceX has had more flights than Blue Origin has had in over 20 years. And I’m being generous by counting Blue Origin’s test flights. Further, SpaceX is far cheaper than all of its competitors and it looks like its rockets are just as reliable, if not more so, than most of its competitors.  So where did Bezos go wrong?  Blue Origin has been rather quiet about their actions, so many things in the early years are speculative, but they did manage to get a small rocket named Goddard to fly to 85 meters in 2006   . And for the next few years, that was kind of it. This is probably a contributing factor to Blue Origin president Rob Myerson leaving the company in 2017  . Bob Smith of Honeywell was brought on board and, well, we’ll get back to him.  In 2006, the United Launch Alliance (ULA), was formed as a joint venture between Lockheed Martin Space and Boeing Defense. Prior to this, there was an a hope of increased competition between competitors that would lower the costs of space launches. Instead, Lockheed Martin Space and Boeing found a way to sidestep competition.    ULA’s Vulcan Rocket: Some assembly (still) required. Source    Later, in 2014, facing the prospect of the upstart SpaceX starting to cut into their market share, ULA decided to gamble and buy BE-4 rocket engines from Blue Origin. Was this to prop up a competitor? Was this to have access to cheaper rockets? Did they think they couldn’t do this themselves? Who knows what their reasoning was, but their gamble failed. The BE-4 engines were to be ready by 2017. The US Department of Defense was looking forwad to ULA’s Vulcan rocket launching with US-made BE-4s instead of the Atlas V rockets launching with Russian-made RD-180 engines.  As of 2021, ULA is still waiting for the BE-4 engines. Of course, it doesn’t help that, with limited experience, Blue Origin agreed to make some of the most difficult rocket engines you can make. The SpaceX initial rockets would never have passed DoD muster.  So, not many flights. Can’t deliver engines. What would you do in Bezos' position?  Well—and this is admittedly before the BE-4 debacle—in late 2011 when Blue Origin was struggling to deliver anything and SpaceX was picking up NASA contracts (thanks, in large part, to the hard work of Gwynne Shotwell), Bezos decided that the only reasonable thing to do was to start working on the New Glenn rocket  . But not a modest version as was earlier planned, but a huge version to rival the Saturn V in height. Suddenly, Blue Origin’s engineers were tasked with creating one of the biggest rockets of all time. Oh, it had to work perfectly the first time. There was no learning curve for Blue Origin. There was a learning cliff and Blue Origin lemmings were being herded over it by Bezos.  Call it a hunch. But maybe, maybe, Blue Origin should have learned to finish something before starting on such an astronomical goal.  So clearly Rob Myerson had to leave, but who’s Bob Smith?  Bob Smith was the man that Bezos picked to run Blue Origin. He has an impressive resume  , including a doctorate in aerospace engineering, a business degree from MIT, and many years of experience building rockets. He should be perfect.  Instead, it’s overwhelmingly agreed by everyone outside of Blue Origin that Smith has been a micromanaging disaster. On Glassdoor, 91% of SpaceX employees approve of the CEO  . For Blue Origin, only 22% approve of the CEO  . For most companies burning cash instead of rocket fuel, having a CEO in place for four years with little to show except demoralized employees and an insanely high staff turnover rate  , the CEO would be out the door.  This Ars Technica article sums things up nicely  :   As Blue Origin has fallen behind SpaceX in recent years, morale has declined, according to multiple sources. Bezos pushed the company further from its roots in late 2017 by sidelining long-time president Rob Meyerson and hiring a traditional aerospace veteran, Bob Smith, to become chief executive. Coming from Honeywell, Smith instituted a more bureaucratic management style, and Blue Origin’s progress seemed to slow significantly. Whereas Bezos' debut flight on New Shepard could have occurred as early as 2019, it slipped by months and eventually years. Critics of Smith’s plodding management style started referring to the company as “Blue Honeywell.”   Jeff Bezos is going to fire Bob Smith. We don’t know when. The only question is why he hasn’t done it yet.  It’s almost as if even Bezos doesn’t believe in Blue Origin’s mission.  I want Blue Origin to succeed. I want Jeff Bezos' company to be one of the most wildly successful companies in history, alonside SpaceX, Rocket Lab, and any other company that’s try to push humanity into space. We need competition here. But where is it? ULA is hanging on with government contracts. Roscosmos wants out of the ISS because they’re so strapped for cash. The ESA and EU are trying to force EU countries to only buy from Arianespace, but the cracks are beginning to show  . Honestly, Blue Origin could have been the one to watch in this space. And now we’re watching them, but not for the reasons they wanted.    Symbolism.   Ironically, part of the issue seems to be that Blue Origin employees know that Bezos won’t stop spending. The money won’t stop arriving. They discuss this in articles, in the Glassdoor ratings, and common sense reinforces this: SpaceX was lean and hungry, Blue Origin has never been either. Like many startups who are flush with cash and fail because they can’t keep their eye on the ball, Blue Origin has no reason to push hard.  And now, Blue Origin’s lawsuit against NASA is destroying their last opportunity to succeed  . NASA, uncharacteristically, has slammed Blue Origin. If you’re unfamiliar with the lawsuit, NASA solicited bids for their 2024 moon lander. Blue Origin’s bid was ridiculously over NASA’s budget and NASA rejected it. So while SpaceX at one point sued the government for the right to compete, Blue Origin is suing because they couldn’t compete.  Here are a couple of tidbits from that article:   Blue Origin was “able and willing” to offer NASA a lower price for its lunar lander but chose not to because it expected NASA to ask and negotiate for a lower price first, the attorneys allege, citing a six-page declaration written by the company’s senior vice president Brent Sherwood in April.   Given that Blue Origin has not demonstrated a track record of being able to repeatedly fly to and from space, why on earth (heh) would Blue Origin think they were a front-runner in the bid or that NASA would come back to negotiate? When NASA, unsuprisingly, said “no”, Blue Origin went to the GAO (Government Accountability Office) and complained it was unfair. The GAO also told Blue Origin “no.” So Blue Origin sued.  Worse, because the lawsuit has halted much of the moon landing program, NASA lawyers wrote the following:   “What begins as a mere procurement delay all too easily turns into a lack of political support, a budget siphoned off for other efforts, and ultimately, a shelved mission.”   We’ve been waiting fifty years to get back to the moon. Blue Origin decided that their pride was more important than the dreams of the American people. It’s not like the fate of their company hangs on this: Bezos won’t stop spending, but they’ve dug themselves a deep hole, lost the goodwill of the American people, and as NASA insiders have said, they’re unlikely to get any government contracts after this stunt.  Seriously, Mr. Bezos, if you believe in your stories about space industry, its time to fire Bob Smith and bring in someone who can fix this damned mess. Everyone already knows he’s gone. Let’s get this done and give Blue Origin a chance to compete.  ","title":"What's Wrong with Blue Origin?","url":"\/blog\/jeff-bezos-1000-problems.html"},{"body":"   Background : I recently proposed a new object system, name Cor, for the Perl core  . This work has been done in coordination with Sawyer X and Stevan Little. What follows are my musings on the process and OO in general. For those things I inevitably get wrong in this discussion, it’s my fault, not theirs.   Contrary to what some might think, the discussions about the Cor OO system are ongoing, but it’s happening quietly, via email, and the holiday period doesn’t help.  And the Moose   has function also doesn’t help. At all. It’s possible that you’ve heard some rumblings about the has function, so let’s clear up some of this. This will take some time, but I promise you, by the time I’m done, that I will have given a rubbish explanation of a hard problem. In throwing this together, I truly understand Blaise Pascal’s comment “I would have written a shorter letter, but I did not have the time.” If I had the time, this would be much shorter and clearer, but I’m too lazy to delete this.  In Moose (and Moo  , but we’ll stick to Moose for this discussion), the has function lets you declare an attribute:  package Point2D { use Moose; has [qw\/x y\/] =&gt; ( is =&gt; 'ro', isa =&gt; 'Num', required =&gt; 1, ); }  The above is a class for the canonical 2D point object that developers have a love\/hate relationship about when learning OO concepts.  As you can see from the has declaration, we have:   Created slots for the data  Created read-only accessors for the data  Required that they be passed to the constructor     That’s a lot to pack into one function, and it leads to confusion like this:  has attr =&gt; ( is =&gt; 'ro', # read-only isa =&gt; 'Int', # declare it an integer writer =&gt; 'set_attr', # but it's read-only? required =&gt; 1, # 'attr' must be provided builder =&gt; '_build_attr', # called if not passed in the constructor );  There are other examples you could put together, but that shows part of the issue: Moose has crammed so much into one function that it’s easy to write code that is confusing or just plain doesn’t make sense. The above does make sense if you realize that required doesn’t mean that the attribute is required to be passed to the constructor (as some assume). The docs say this  :   Basically, all [required] says is that this attribute must be provided to the constructor or it must have either a default or a builder. It does not say anything about its value, so it could be undef.   But what does declaring something as read-only and providing a writer (mutator) mean? That’s confusing because we’re really packing a lot of meaning into a single function.  What’s worse, it’s harder to write code that does what we need. If I want a private, internal slot, per instance, with no accessor, do you know how to do that in Moose? We tend not to do that too much with Moose simply because it’s hard to do, despite being trivial in other languages.  Well, you can declare a slot as bare (instead of ro and friends) and then do:  my $value = $self-&gt;meta -&gt;get_attribute($attribute_name) -&gt;get_value($self);  Except no one does that (by “no one”, of course, I mean “the set of no developers who aren’t not Stevan.“) They just declare the attribute private and that’s that. And I don’t blame them. But it means that everyone just makes accessors for everything, and that makes it much easier to make public accessors for everything and when I’m cleaning up code for a client, that’s a problem because it makes it much harder for me because now I have a public contract that I have to respect, even if there wasn’t originally a need to expose that data.  Encouraging people to write accessors is a bad idea, but people do it all over the place, just as they often make attribute mutable for no good reason ( and here is a great example of why mutable objects are dangerous  ). A good rule of thumb for OO design: make your interface as small as possible. Moose, unfortunately, offers an affordance to making our interace as large as possible.  Moving along, what does that point class look like in Java?  public class Point2D { \/\/ slots private double x; private double y; \/\/ accessors public double x() { return x; } public double y() { return y; } \/\/ constructor public Point2D(double x, double y) { this.x = x; this.y = y; } }  Note how the slots, the accessors, and the constructor arguments are all nicely decoupled, leaving no ambiguity.  And without going into detail, private slots with no accessors are trivial in Java, at both the class and instance level.  But where does that leave Cor ? Well, Cor aims to be core , meaning that overloading has is problematic. We’ve learned a lot from Moose about how to make OO that feels “perlish”, but we need to keep growing and learn from our mistakes. Returning to this:  has attr =&gt; ( is =&gt; 'ro', isa =&gt; 'Int', writer =&gt; 'set_attr', builder =&gt; '_build_attr', required =&gt; 1, );  For Moose, that’s OK. For Cor , that’s not, because it means you can write code that is, at best, confusing.  And then there are things like this:  my $auth = Authenticate-&gt;new($one_time_token); # or my $auth = Authenticate-&gt;new( user =&gt; $user, pass =&gt; $pass ); # or my $auth = Authenticate-&gt;new({user =&gt; $user, pass =&gt; $pass});  Here’s one way you could try to handle that:  around BUILDARGS =&gt; sub ($orig, $class, @args) { my %arg_for = @args &gt; 1 ? @args : 'HASH' eq ref $args[0] ? $args[0]-&gt;%* : ( token =&gt; $args[0] ); return $class-&gt;$orig(%arg_for); };  In languages with multi-dispatch, if we add a new way of instantiating an object, we can just add a new constructor and the language will handle the dispatching for us. With Moose, we need to use BUILDARGS for this, and manually handle all of that dispatching ourselves. That means more bugs. If Perl had something like multidispatch, we could possibly write:  method BUILDARGS (Code $orig, Class $class, UUID $token) { ... } method BUILDARGS (Code $orig, Class $class, Str :$user, Str :$pass ) { ... } method BUILDARGS (Code $orig, Class $class, HashRef $args) { ... }  For Cor , we’re not going to get there because Perl doesn’t yet have robust signatures like that and we’re probably never going to get multidispatch, but decoupling the very overloaded meaning of has will help, and limiting how we can pass arguments can help (no more “named pairs” or “hashref” decision making).  And here’s a fun one! An attribute that you can’t set, even though it looks like you can pass it in the constructor:  #!\/usr\/bin\/env perl package SomeClass { use Moose; has 'username' =&gt; ( is =&gt; 'ro', isa =&gt; 'Str', required =&gt; 1, init_arg =&gt; undef, default =&gt; 'Bob', ); } # this prints &quot;Bob&quot; print SomeClass-&gt;new( username =&gt; 'foo' )-&gt;username;  Of course, Cor could try to trap every possible illegal (or confusing) combination and then what?   Warn about them?  Ignore them?  Make them fatal?  Do this at compile-time?  At runtime?   And if we check illegal combinations, then if we ever extend has to have more functionality, we have to figure out the new “illegal” combinations (such as making attributes simultaneously required and lazy).  Do we stop developers from shooting themselves in the foot or hand them a gun?  The counter-argument I hear from many developers is “we don’t need to separate slots and attributes or a different syntax for declaring attributes. What we’ve been doing works.”  And frankly, this has worked fairly well. If the new Cor proposal doesn’t provide some dead-simple way to create accessors, then we’ll wind up with 42 different modules to provide that and we won’t be that much further along than when we started.  There are so many decisions to be made with Cor , many of which would be dead wrong if we move too quickly and we don’t want to get this wrong. And the lack of multi-methods and the inability to introspect signatures means the BUILDARGS and BUILD pain will remain in some form (though BUILD isnt' too bad).  Thus, we really want to have some separation of slots, attributes, and constructors, but making an easy syntax for that is is a challenge. As the old line goes, “making things hard is easy; making things easy is hard.” And making a solution that is easy and does the right thing and that all Perl developers will like is impossible. And there will be developers looking at the final solution and sayind “no” because their one pet peeve wasn’t respected.  As for our initial work with the syntax, the core Perl devs liked what they saw in the initial proposal, but it’s one thing to say “shiny!“. It’s another thing to hammer that shiny onto something that already exists. There are a number of approaches which can be taken, but they require hard decisions. For now, Cor will be focusing on the syntax.  ","title":"Moose \"has\" a Problem","url":"\/blog\/moose-has-a-problem.html"},{"body":"    A Brief History of Private Spaceflight  The Four Phases of Market Evolution  Spaceflight is Entering the Stability Phase  Starship Changes Everything  Costs  Disruption  What the Future Holds  New Industries  Mining in Space  Disruptive Uses?  Investing  Be Patient     A Brief History of Private Spaceflight  At the end of the 20th century, spaceflight was considered a “national” endeavor, limited to a few countries such as the United States, Russia, France, China, Japan, and so on. Companies like Boeing, Honeywell, Roscomos, Arianespace, and others provided space offerings that were attached to national programs, not private programs.  But at the dawn of the 21st century, things started to change. Jeff Bezos founded Blue Origin  in 2000. Elon Musk founded SpaceX  in 2002. Richard Branson founded Virgin Galactic  in 2004. Blue Origin was in stealth mode. Elon Musk was laughed at. Virgin Galactic was viewed as the toy of a billionaire playboy. Bezos, Musk, and Branson, serial entrepreneurs all, saw a future for spaceflight that others had not. They wanted to create new industries.  At that time, VCs weren’t interested in space startups. Investment funds didn’t carry them in their portfolios. By 2014, when Google acquired satellite company Skybox Imaging  , that changed. The era of private spaceflight was starting to mature.    The OTRAG Rocket, 1977   But just as many people aren’t aware that the internet started long before the mid-90s, private spaceflight began long before this century.  Commercial satellite launches began in the 1960s, long before humanity landed on the moon. The first amateur satellite, OSCAR I, was launched in 1961  , a scant four years after Sputnik, and the first commercial satellite, Telstar 1, was launched in 1962  .  And then there was OTRAG. OTRAG was a private rocket company founded in 1975 by Lutz Kayser, a German entrepreneur. The intent was to build low-cost rockets an order of magnitude cheaper than existing rockets. Amazingly, Kayser succeeded, but the history is tragic  .   Kayser’s activities made the great powers nervous. The USSR and France were not interested in Germany achieving an indigenous long-range rocket activity. American rocket makers were not interested in having a low-cost competitor. A propaganda campaign began, alleging OTRAG was a cover for German and South African nuclear cruise missile development. Crude Soviet-source disinformation was eagerly picked up and given credibility by the American mainstream media. The government of the Congo, [where OTRAG was launching from], was pressured by the Russians to withdraw permission to use the site. OTRAG left the country in April 1979.   They relocated to Libya, but the Libyan government seized their rockets in an attempt to create their own rocket program. Fortunately, without Kayser’s experience, they failed, and the dream of private spaceflight failed with it.  If Kayser succeeded, we might have our moon base by now. We might be on Mars already. Instead, we’re still waiting, decades later.  There’s a decent Wikipedia page that sums up the history of private spaceflight  , but mostly, the nascent industry sputtered along in fits and starts, usually followed by bankruptcy.  It wasn’t until the 21st century that the private rocket industry exploded.    Now, everything is changing, but what’s going to change? To understand that, we have to know a bit about the evolution of markets.  The Four Phases of Market Evolution  Note : much of the following is derived from the work of Simon Wardley  , but I’ve adjusted some of the terminology.  Markets (and new products within them) tend to evolve through four maturity phases. Each has unique characteristics and should be treated differently. This is boring for most, but it’s important. These phases are:   Novelty Phase : A new market emerges. Unknown how\/if consumers will respond.  Stability Phase : The new market is stabilizing, but not necessarily mature. Still experimenting with different approaches.  Commodity Phase : Robust competition amongst competitors. Less product differentiation due to market maturity.  Utility Phase : Market is mature, widespread, very stable, and margins are thin. New markets are built on top of these.    Note : the above descriptions are far too brief. For extended descriptions, I have a brief article which explains them in more detail, including examples .   For private spaceflight, the “novelty phase” was when entrepreneurs started to see potential, but it’s very high-risk, high-reward. As first movers, if they succeed (most did not, of course), they can reap tremendous benefits. This is why Elon Musk regularly tops the lists of the world’s richest people.  As we move from novelty to stability to commodity to utility, we see risk gradually decreasing:    Novel markets are very high-risk.   Why does risk decrease? There are various factors, including customers becoming more comfortable with the market and having new ideas how they can utilize the market. But to succeed in this evolution, the products in that market must be reliable . Would you buy a cell phone that only has service 50% of the time? Probably not.  What this means is that as markets mature, their products become more stable:    Stable markets are very predictable.   Now compare this with air travel  :   In the industry’s embryonic era, accidents were disturbingly numerous. Ironically, 1929, the year of “The Great Crash,” was also one of the most crash-ridden in aviation history, with 24 fatal accidents reported. In both 1928 and 1929, the overall accident rate was about one per every million miles flown. In today’s system, an accident rate of that magnitude would result in a nearly incomprehensible 7,000 fatal accidents each year.   By 2017, there were no recorded deaths of passengers in commercial passenger jets  . In 2019, a National Safety Council’s recent odds-of-dying data  reports that you have a 1 in 107 chance of dying in a car crash. Airline passenger? “Too few deaths in 2019 to calculate odds.”  But what about spaceships? Astronaut Chris Hadfield reports that in 1981, you had a 1 in 9 chance of dying in a space shuttle launch. By 1995, the odds of dying had already dropped to 1 in 38. Of course, those are per flight, not per lifetime, but if this trend continues, will we see ourselves in the bizarre position of seeing our odds of dying on a spaceflight lower than our odds of dying in a car crash?     Today, no astronauts have died since the 2003 Columbia space shuttle disaster  . (Note: the last cosmonauts to die in space were in 1971  ). Though in one bizarre incident, an astronaut almost drowned on a July, 2013 spacewalk  .  That doesn’t mean spaceflight is suddenly safe, but it’s far safer than it ever has been.  Spaceflight is Entering the Stability Phase  Long viewed as a novelty, spaceflight is entering the stability phase. It’s not there yet, but it’s on the verge and SpaceX is leading the way. While the novelty phase offers higher rewards, its risk level is too high for all but the most informed (and wealthy) investor. However, when the stability phase is entered, the market has begun to solidify, the products are becoming trusted, more investors can understand the market, and the risk has dropped tremendously. Of course, the reward has dropped, too, but that seems a fair trade-off.    Advanced Blue Origin Lander Prototype.   However, particularly if we’re viewing things through a capitalist lens, it’s hard to argue that a market is “stabilizing” if there’s only one real supplier. Blue Origin, despite their recent suborbital tourist flights, is struggling . Due to poor performance, investors worry about Virgin Galactic  and the new lawsuit against Richard Branson and Virgin Galactic chairman Chamath Palihapitiya over allegedly dumping shares while covering up spaceship defects  is not going to help matters.  Roscomos, the state-owned Russian space company has long struggled financially and recently announced steep budget cuts  are not going to help. Private Russian spaceflight company Dauria is fighting bankruptcy  . French company Arianespace reported a 30% increase in 2021 revenue  but their lack of reusable rockets and their promised reusable “Prometheus” engine (to be first launched eight years from when I’m writing) being delayed means they’re falling further and further behind SpaceX. Worse, the impending arrival of Starship means Arianespace is in serious trouble  .  So with them not going anywhere, what happens if SpaceX collapses? Elon Musk has already warned his employees about potential bankruptcy if it can’t speed up production  (though he later admitted this was unlikely). But I’ve talked to plenty of people about the space industry and most of them are unaware of the numerous successful competitors out there.  Virgin One was spun off from Virgin Galactic and shortly after its first successful launch of seven satellites  announced that NASA put them on their short list of satellite launch providers  . This doesn’t guarantee they’ll be selected for launches, but the private sector is watching them. They can’t launch very large satellites, but their launches are only $12 to $15 million per launch  . They’re a force to watch.  Rocket Lab and Firefly are also looking very competitive  :   Rocket Lab’s small Electron rocket goes for about $7 million per launch, Firefly’s medium-sized Alpha rocket goes for $15 million per launch, and SpaceX’s Falcon 9 is advertised at $62 million per launch.   There’s also UP Aerospace  , United Launch Alliance  , SpaceForest  (services starting in 2023), and many others starting development. Raphael Roettgen  , on the TerraWatch podcost  , estimated that there are around 200 launch startups around the world today.  The market is booming and stabilizing. Customers can now choose between a wide varieties of rockets with different payload sizes and price points, but there’s still an enormous amount of risk and it will be a couple of decades before we can approach the commodity stage.  Starship Changes Everything  A few years ago, a friend in Nottingham told me she wanted to spend the weekend in Ireland, so she bought a plane ticket. When she changed her mind, she threw away the ticket. It was only £20. No longer the domain of the wealthy, air travel is for everyone and this is (eventually) coming to space.  Costs  For the space shuttle, the average cost per launch for the last decade was around one billion dollars  . For Starship, Musk expects launch prices of less than ten million within two to three years  . Eventually, he projects those prices could be less than a million dollars a launch.    In contrast, the Concorde’s inaugural 1973 flight from Paris to Rio de Janeiro  , roughly 5,700 miles (9173 km), operating at a cost per flight of up to $5.20 per mile  , could be looking at just shy of $30,000 for that flight. Adjusted for inflation, that’s almost $190,000 today. However, the $5.20 figure was for a 1972 estimate. In reality, the Concorde was subject to wild cost overruns, so a $190,000 flight cost is certainly low.    That Starship may not be significantly more expensive than a Concorde is mind-boggling. The fact that Starship might actually cheaper than planes for long-haul flights is even more mind-boggling  .  But wait, there’s more! The cost of building the Boeing 737  , one of their least expensive aircraft, is just over $100 million. At the high end, Starships could cost around $37 million each  , but an optimistic mass production rate could drop that price to $5 million per Starship. This is insane .  Disruption  The lower costs, however, have a knock-on effect for the entire industry. The media tends to think of the cost of the launch, but the insanely high launch costs ripple through the industry. Want to make a lunar lander out of stainless steel? Too bad. Titanium is about 35 times as expensive as stainless steel  , but it’s weighs half as much. With costs per kilogram in the thousands  , every penny counts.       Launch Vehicle  Payload cost per kg      Vanguard  $1,000,000    Space Shuttle  $54,500    Electron  $19,039    Ariane 5G  $9,167    Long March 3B  $4,412    Proton  $4,320    Falcon 9  $2,720    Falcon Heavy  $1,400    Starship  $100      Yes, Starship costs can potentially cost an order of magnitude less per kilogram. What else does that mean?  It means missions can be run in parallel, not in series.  Think about a string of electric lights. If they’re run in series and one goes out, they’re all out. If they’re run in parallel and one goes out, the rest keep working and you can compensate for the missing bulb.  Missions in space today are long series of single points of failure once launched. Consider a simple (ha!) robotic rover to Mars. It’s been devilishly hard to land on Mars, so we have to:   Design the rover  Order the parts for the rover  Ensure all tolerances are perfect for the rover  Assemble the rover  Test, test, test, and retest the rover  Return to step 2 if the parts aren’t up to snuff  Return to step 1 if the design isn’t up to snuff  Do all of this in a near-sterile environment so we don’t contaminate Mars  Move the rover to the rocket  Have a perfect launch  Wait six to nine months while the rocket heads to Mars  Enter the Martian atmosphere at just the perfect angle  Survive the descent to the surface  Scream in agony as the dust storm covers the solar panels and kills the mission   I skipped a few steps in there, but prior to launch, costs are astronomically high to due having to be perfect , absolutely perfect , because once you launch, any failure at any single point renders all of the previous work moot.  With Starship, you could build multiple rovers, with much less costly materials (weight no longer being the major consideration), lower tolerances for many of the components, and fly all of them to Mars on multiple launch for less than the cost of a single rover today. A couple of flights fail? Fine, you still have a fleet of rovers on Mars. And no, you can’t reduce all of the tolerances, but by building a fleet, the individual costs can be lowered as you gain more experience in procurement and assembly.  Heck, the Perseverance rover to Mars will have a projected final mission cost of almost three billion dollars  . Musk’s long-term goal is to make a passenger ticket to Mars as low as $100,000  .  Starship changes everything.  What the Future Holds  Just as the internet has proven to be a hugely disruptive force for industry and society, so will the new space industry. But like the internet, we don’t know what that impact will be.     New Industries  As we improve our rocket technologies, many of the new industries are already on the horizon. Space tourism on rockets and even the ISS have already happened. But space hotels are being actively planned  :    Orbital Assembly Corporation space hotel concept.   Real-time satellite imaging is widely available, high-speed, low-latency internet  is starting, and Japanese company Astroscale  is already planning to remove space debris and provide on site inspection and maintenance of satellites. ( Dealing with the ever-increasing threat of space junk is a serious problem  ).  Mining in Space  Of course, no story would be complete without discussing mining. If you are at all interested in space, you’ve long heard that the world’s first trillionaires will be space miners. The early space mining companies are gone  because they, like OTRAG, were early adopters who adopted too early. They generally anticipated that the first space mining would involve mining extremely valuable ... water.  Yes, water. They were going to sell it to spacecraft using various means of water propulsion  (seriously, the idea is picking up steam). However, the industry they wanted to support wasn’t ready to be born, so the space mining companies drowned (forgive me). But water is going to be a seriously valuable resource in space. I describe this briefly in one of my public presentations  (if the video doesn’t start at the correct spot, jump to 50:13):      Side note: I’m also a public speaker, so if you want to have me give a talk about this, let me know !   But what else might we mine? We are facing a near-term global lithium shortage, so that would be awesome. But where do we find it? Lithium dissolves easily in water and is found in salt flat brines like those of Chile, or pegamites , a type of rock that forms in water-rich magmas. It’s estimated that the moon may have the same relative amount of lithium as Earth, but without water, it wouldn’t be concentrated enough to efficiently mine. Mars? Maybe, but we’re not sure. Ceres has been speculated as a source of minable lithium, but we can’t send our mining robots on a hunch. We need to go prospecting to find what we need and currently, that’s too expensive—until Starship is fully operational.  The same goes for Rhodium and the other platinum-group metals. Without a low-cost ability to to prospect space, we can’t send out mining equipment, much less miners.  But let’s say we find a near-Earth asteroid with the metals we want. The cost, per load, of those materials needs to be lower than the cost of building our prospectors, launching, searching, building our mining robots, launching them, mining the metals, refining the metals on site, and then returning them to Earth. As it stands now, we can’t meet those costs and even with Starship, getting them back to Earth is difficult. We can’t use parachutes. Simply crashing them on our planet will destroy much of it in the atmosphere and needlessly endanger lives, so they need to be brought back in a rocket. It’s simply not economically feasible.  More accurately, asteroid mining isn’t economically feasible until we have an established industry in space . At which point, all of that changes because we can build, launch, and return in space. That’s much less expensive and we can use those materials to build out space-industry in a more cost-effective manner.  Sadly, that’s decades away. But asteroid mining might still be viable.  The moon is regularly hit with asteroids and probably about 20% of those are metallic. Building stations on the moon and using the materials there might be worthwhile...and we’re already looking at lunar stations. Thus, we can do asteroid mining, so long as we’re doing it on the moon.  The south pole of the moon is a potential site for setting up our stations  . It appears to have abundant water and if we send out plenty of robotic prospectors, finding a location near silicon and platinum deposits would give us the ability to mine the materials to make solar panels to power our lunar outposts. But we won’t know until we start prospecting.  Disruptive Uses?  We’ve already discussed Starlink  , a mega-constellation of satellites to bring high-speed, low-latency internet connections to the entire planet. There are considerable challenges involved, including efforts to reduce their visibility  and not ruin ground-based astronomy. The long-term goal is to have over 30,000 satellites operational, so effort is being put into ensuring defunct satellites de-orbit as quickly as possible to mitigate the problem with space debris  .  But it’s possible that Musk may be planning to become a trillionaire by creating space-based solar power generating stations  . These generate nice, clean solar power in space and beam it back to Earth. Before you think this is just science fiction, China is already planning on doing this  , but unlike SpaceX, they don’t have access to Starship. That being said, the the long-term benefits outweigh the costs of launching via China’s Long March 5  rocket. China is pushing forward.  Musk’s path may be less clear because if successful, he threatens the fossil-fuel industry and they will do everything they can to stop him (unless he creates a partnership with them, which I doubt). They will be lobbying like made, citing environmental concerns of the launches (oh, the irony), trying to find regulatory means of stopping him, and pointing out to politicians all the jobs their districts stand to lose if these stations are operational. This seems strange, but we’ve already seen this with Tesla Motors. This will be OTRAG all over again: political and market forces working hand-in-hand to block innovation.  There will be other disruptive markets to emerge from this industry sometimes being referred to as NewSpace  , but like Black Swan events  , we can’t predict them in advance, and they will have enormous impacts on markets.   Note : space elevators will not be the disruptive force  . Not only are they probably decades away, but an ascent to space will likely take weeks.   Investing  Like OTRAG, early space mining companies died because they were too soon. Early investors in this “novelty” stage of the market lost a lot of money. Now that we’re entering the stability phase, investing presents less risk, though less reward.  Before NewSpace started entering the stability phase, there were few, if any, VCs or financial services company offering services. Today, there are many of them. Today, investment giant Morgan Stanley is even offering space investing  . However, space investing today is mainly in the hands of the wealthy. Sadly, this is probably a good thing because as if often pointed out by investors, unless you put in 10,000 hours or more in deep study of NewSpace, you won’t have the expertise to evaluate investments.  NewSpace can’t do the early internet thing of “throw money at the wall and hope it sticks.” The amounts of money involved means you have to have an extremely high chance of a payoff. These companies aren’t investing in new technologies so much as they’re investing in markets with solid business plans. This sets this industry apart from the early days of the tulipmania  Internet Bubble.  Of course, other companies are now getting involved. The highly-regarded International Space University  (ISU), has a Master’s Programs in Space Studies and offers a wide variety of professional courses in different aspects of the space industry.  Be Patient  We’re at the NewSpace equivalent of the internet in the mid-90s. It’s booming tremendously, but due to the extremely high costs and high risks, it’s being approached in a healthier manner than the internet boom. However, we won’t see internet speeds of advancement. For much of the internet, it was a matter of building ISPs and connecting people and letting the industry blossom. For NewSpace, the entire industry needs to be boostrapped, including the core infrastructure.  It’s not going to impact us directly today–no one is buying a cheap flight to an orbital hotel–but it’s coming and despite the naysayers, it’s not stopping. And even if SpaceX unexpectedly goes bankrupt (it won’t), it will still keep coming. I don’t know if I’ll live long enough for a vacation in space, but my daughter will. So will your children. So will many of you who read this.  The long-awaited promise of a future in space is finally arriving.  ","title":"Time to Invest in Space?","url":"\/blog\/time-to-invest-in-space.html"},{"body":"  In 2018, the documentary Amazing Grace  featuring Aretha Franklin was first released. It was filmed in 1972. You might wonder why it took almost five decades for a documentary of the top-selling gospel R&amp;B album of all time to be released. Sure, Franklin sued twice to prevent its release, but we’ll skip that. The real reason is hilarious.     Sydney Pollack  , now known as one of the most influential filmmakers in history, was approached by Warner Brothers to film Franklin’s concert in a church. He agreed immediately upon hearing her name. The camera crew set up and by the second night, word of her electrifying performance had spread so quickly that even Mick Jagger was in the audience. This was a legendary event.  Warner brothers didn’t want to wait five decades for the film to be released, but to understand why it took so long, we need to understand a bit about how a movie is made.  You know how you take 100 pictures and maybe one or two are any good? That’s a documentary. You see maybe an hour or two of something amazing, but behind the scenes (pun intended!) are many more hours of footage that never leave the cutting room floor. Some of that footage is awful. Some may be amazing, but it doesn’t fit the director’s vision. For whatever reason, you shoot tons of footage and use almost none of it. For Amazing Grace, twenty hours of footage became 87 minutes of film. In other words, less than 10% of the footage was used.  A film camera, as you know, records images, not sound. That’s for the microphone. The film and the sound for the film are thus recorded separately.  Later, when putting together a rough cut of the film, the “sync sound” (the actual sounds recorded while filming) gets synced with the film. You’ve probably seen films where the sound doesn’t quite match the actor’s lips. This is sometimes known as “lip flap” and it’s annoying as hell, so you need to get it right. With twenty hours of film and twenty hours of sound recording, how do you get it right?  Typically, we use a clapperboard  , also known as a “film slate.” There are various styles, but they usually include at least the date, the production, the scene and the take. But the one thing they always include is that long bar across the top. You’ve all seen video of someone yelling “lights, camera, action!” And then the second assistant camera  holding the slate slaps the top down with a sharp “clack.”    A film clapperboard Source    In the rough cut of the film, you’ll see that clapperboard being clapped. In the audio recording, there’s a sharp spike when that clapperboard makes that noise. You can then use those to sync the film and the sound.  Sydney Pollack forgot to bring clapperboards to the church. After, they discovered they couldn’t sync the sound to the film. They shot the world’s greatest gospel documentary as a silent film.  By June of that year, Aretha Franklin’s album had sold millions. The movie would have been a slam dunk. Via Hollywood Reporter  :   The choir director from the Watts recordings was brought in to try to lip-read the reels, but after months of work, only about 150 minutes of footage had been matched with sound, none of it adding up to a complete, useable song.   It wasn’t until 2008, using computers to synchronize the film and the sound, was the movie finally complete. Aretha Franklin sued (twice) to prevent the release of the film. After she passed, an agreement was reached with her estate to release the film. It grossed less than $6 million dollars, a far cry from what it could have been, but Metacritic rates it 94 out of 100  , or “Universal Acclaim.”  Of course, Aretha Franklin deserves nothing less.   If you’re interested in the film industry, Castaldo Academy  has produced a short video explaining the clapperboard’s use.     ","title":"The Documentary That Took 46 Years to Release","url":"\/blog\/the-documentary-that-took-46-years-to-release.html"},{"body":"    Introduction  Forget Europe  Working Holidays  Teaching English  Volunteer Work  Au Pair  Studying Abroad  Joining the Military  French Foreign Legion  Foreign Service Exam  Fantasy versus Reality  Conclusion     Introduction  Constantly on Reddit’s IWantOut community  , or if they’re a bit clued in, in expat forums, I see variations of the following question (the following was a real post on the now-defunct “Yahoo Answers” forums):   I would mainly like to move to Germany but I am wanting to move to Europe and learn fashion design. I don’t know what sort of job I can get over there being so young and not speaking another language. I don’t feel at home in the USA and finally did in Europe and I have to go back ! what should I do?   I call these fairy tale posts.  To figure out how to move to another country, you have to take stock of what you skills you possess which might help to make this happen. Here’s the typical profile of people making fairy tale posts:   No life experience  No work experience  No education  No language skills  No money   This does not bode well for your chances, but you might have the one skill necessary to pull this off: determination.  To start with, read Why you’ll say “no” to living abroad . I consider this the foundation post of the expat posts because it’s critical to understanding the process. You have to make sure that you can seize any opportunity which comes along (this doesn’t mean you should take any opportunity).  The very first thing you need to do is apply for your passport. If you don’t have a passport, you’re not serious about moving abroad. If you can’t be bothered to get a passport, you may as well stop reading.  The second thing you need to do is take stock of your life and stop doing anything which might land you in legal trouble. Of course, that’s a very personal decision, but if you wind up with a criminal record, your limited chances of living abroad are approaching zero. Maybe you don’t think the government should outlaw drugs, but your personal beliefs are meaningless when another country sees you have a drug conviction. You need to decide what’s more important to you.  The third thing you need to do is set your life up so you can leave as soon as is feasible. Long-term leases or owning pets might be an obstacle. However, working towards a college degree is a benefit, even if you don’t get to leave tomorrow. Always ask yourself, “what’s stopping me from leaving now?” As agonizing as this may sound, it also means considering your partner, if you have one. Would you leave a girlfriend\/boyfriend for a chance to move to New Zealand? You have to understand your motivations here and you should be prepared for tough choices.  If you do have serious work skills, and by “serious” I mean “highly skilled labor”, I’ll later be sharing a “How to get a work permit” series, but if you’re young and unskilled, you probably don’t have a chance at a work permit. So what do you have?  Forget Europe  I don’t really mean that, but if you think “where can I move” instead of “I want to move to Europe”, you’ll have many more opportunities. The biggest mistake I see many would-be expatriates make is forgetting that the world is round. For example, Montevideo is a large, cosmopolitan city and Uruguay is relatively easy to emigrate to, basically requiring around $1,500 a month income and a lot of paperwork  . Can you find a remote job that pays more than $1,500 a month? Maybe you can get to Uruguay. However, one person who could have done this told me “I don’t want to live in the sticks” (US slang for “middle of nowhere”). Montevideo is not “the sticks”, but if you don’t do your research, how would you know?  Working Holidays  For young people, you might want to consider working holiday schemes  . Australia and New Zealand offer them and while they aren’t designed to allow you to permanently live there, once you’re there, your options will open up tremendously.  Teaching English  Many young people complain that they don’t want to teach English abroad  . Too bad. It’s one of the most popular ways that young, unskilled workers can move to another country. Many people report heading to countries with a strong demand for English teachers and finding work under the table, but this is very risky. You might not find work and, if you do, you’re probably an illegal immigrant and risk deportation. Being deported from a country means you probably can’t get back in for many years. In fact, some countries won’t let you in if they find out that you’ve been deported from another country.  Volunteer Work  Many people are surprised to find out that volunteer positions around the world often require you to pay for the privilege of being a volunteer. Sadly, this is not a scam. Many volunteer organizations are poor and simply can’t afford to fly you to Paris, put you up in a flat and feed you while you work for two hours a day baby sitting someone. Do your research as some “volunteer” organizations are scams, but if you find a reputable volunteer organization, you can at least have a short time in another country and gain valuable life and work skills. Plus, it looks great on your résumé\/CV.  Au Pair  An au pair is basically a young person (usually a woman) who travels to a new country, stays with a host family and looks after the children for a time. It’s usually a short-stay trip and an au pair is not a nanny. Families do this to expose their children to other cultures and, sometimes, because they enjoy giving people from other countries an opportunity to see the world. Be careful, though. While it’s rare, some au pairs find themselves horribly abused by their host families. Work through a reputable agency and be safe.  Studying Abroad  Studying abroad offers you not only the chance to live in another country, it allows you to earn a degree which makes staying abroad much easier. In fact, many countries give preferential immigration treatment to students who have graduated from that country’s universities.  Many foreign universities are eligible for US federal financial aid. Other countries, such as Germany and Norway, have free university programs which foreigners are eligible for. This probably means staying in the US for a couple of years, going to university and earning good enough grades to qualify for foreign universities. If you can do this, it’s one of the best ways of getting out and staying out.  Joining the Military  This one is controversial, but yes, the military will often send you abroad. It’s one of the easiest ways of gaining life skills, work experience, seeing the world and saving money for college. Of course, you might have to kill the foreigners you meet. Your own values\/political beliefs will have to be examined strongly before you consider this.  French Foreign Legion  Sorry ladies, this one is for the men only.  I used to run a very popular (now defunct) expat blog and every week, searches for “French Foreign Legion” showed up in the list of most popular searches on the site. I did quite a bit of reading about them and learned that many people think the Foreign Legion of today is the Foreign Legion they saw in black-and-white movies, or read about in Soldier of Fortune magazine. In particular, there are three myths I want to dispel about the modern Legion:   They no longer accept felons  The no longer grant French citizenship under new identity  Most applicants are rejected   To join them, you need to be a young man, in good health, and buy a plane ticket to France. And be willing to kill people.  Here’s their web site  .  Foreign Service Exam  This is not as widely discussed and I should write more about it, but if you are interested in the US Foreign Service, the qualifications are:   U.S. citizens on the date they submit their registration package  At least 20 years old and no older than 59 years of age on the day you submit your registration  At least 21 years old and not yet 60 on the day you are appointed as a Foreign Service Officer   Note that neither foreign language skills nor a college degree are required. However, very few people pass their selection criteria. Prepare to do a lot of reading about being a foreign service officer  and accept that very few people pass the entrance exam. It takes a lot of study and discipline, but it’s one of many strategies available to you.  Fantasy versus Reality  When people as me what it’s like to be an expat, I often reply “how do you feel about immigrants”? It’s hard being an expat. Don’t underestimate homesickness. Many people want to be expats so they can tell their friends what wonderful and exciting lives they have but their friends are thousands of miles away. And the expat life is not glamorous. The French have a saying, « metro, boulot, dodo ». That means “subway, work, sleep”. It’s a typical “day in the life”. Being an expat means that you’ll get up, go to work, go home, have dinner, and sleep. That’s the majority of your existence at home and that will be the majority of your existence abroad. It’s the things outside of « metro, boulot, dodo » which make being an expat so rewarding, but those aren’t the bulk of your existence.  It’s not a magical, fairy-tale world out there, but it’s an incredible one. Immersing yourself in other cultures and understanding how other people see the world is one of the most wonderful things you can do in life.  Conclusion  If you’re young and determined, you have multiple avenues to live abroad, either permanently or temporarily. I would strongly recommend that you pursue several at the same time, maximizing your flexibility. Living in another country is not a right, it’s a privilege that you have to earn. If you’re determined enough, you can do it.  And keep reading this blog. As I research more ways to get out, I’ll post them here, along with my personal experiences. I’ve helped others leave their countries and I want to help you, too.  ","title":"Young Person's Guide to Moving Abroad","url":"\/blog\/young-persons-guide-to-moving-abroad.html"},{"body":"    The Continuum of Everything  What Is Life?  Life as Biopresentation  Objections to Biopresentation  Why is rethinking life important?  Is Biology the Continuum’s End?     The Continuum of Everything  I recently wrote about searching for extraterrestrial life in our solar system . In that post, I put forward a novel way of defining life that I call biopresentation . So it was with some surprise when I listened to a recent podcast of Isaac Arthur entitled “Post-Science Civilizations” and realized that part of what he was explaining reached right into the heart of my argument and even expanded it. Here’s the YouTube video for it:     In this, he discusses what would happen if we “run out of science,” though there are several possible paths to that. As someone who loves chess, one of the things I appreciated was him pointing out that knowing all the rules of chess doesn’t mean you’re a master of it. So if we learn all the rules of physics, that doesn’t mean we can instantly figure out what to do with them.  However, my big “a ha!” moment was Isaac Arthur pointing out that chemistry is emergent from physics and biology is emergent from chemistry. For now, we’ll avoid the obvious, and unanswerable, question: “what, if anything, is physics emergent from?”  Since chemistry emerges from physics, chemistry is physics. Since biology emerges from chemistry, biology is chemistry, but it’s physics, too. Now for some, this seems trivial. After all, we’ve learned from astronauts in microgravity that our body needs gravity to function well. But it turns out that life might not only require quantum mechanics to lead to chemistry, but life might be actively requiring quantum mechanics to continue existing. There’s an entire field called quantum biology  . We’ve known for over a decade that photosynthesis might rely on “quantum coherence” to generate energy  . Quantum neurophysics  is a new field, too. Special relativity may be critical to our functioning, not just to our existence.    Getting back to the main point, the question we should ask is, “if biology is emergent from chemistry, where do we draw the line?”  We don’t. It’s a continuum. Rather than refer you to a previous post I wrote, I’ll repeat myself.  What Is Life?  This must be prefaced with the “life as we know it” caveat (and I’ll give examples later). And I’ll come right out and say what’s on my mind: most definitions of life are a steaming pile of shit.  One definition of life  , as taken from Merriam-Webster, is “an organismic state characterized by capacity for metabolism, growth, reaction to stimuli, and reproduction.”   But right off the bat we see a problem. Mules, for example, don’t reproduce.   Would you say they’re not alive? Of course not. Biologists get much more sophisticated than Merriam-Webster, but many of their definitions still boil down to a true\/false value.  What about fire? It reacts to stimuli and grows  . We could even argue that it metabolizes—what it consumes for energy to keep growing—and reproduces by budding, but we don’t think of it as “alive.”  There are other properties of life, however. Homeostasis  is the body’s ability to internally regulate itself and keep functioning. Does fire exhibit homeostasis? It’s hard to argue that it does. However, warm-blooded and cold-blooded animals have completely different approaches to homeostasis and homeostasis evolved over time  , so there are different levels of homeostasis. Maybe we can argue that humans have a homeostasis level of ten while fire has a homeostasis level of one? A piece of flint, however, would have a homeostasis level of zero.  But why would we argue that?  Life as Biopresentation  When I was working on ETL  systems to reduce the cost of Phase III clinical trials  , I often found myself working with data supplied by many pharmaceutical companies. Company A might refer to “Dr. Robert Smith, St. Mary’s Hospital, Arlington, Virginia.” Company B might refer to “Dr. Bob Smith, St. May’s Hospital, Arlington, Virginia.” Are they the same person? “St. May’s Hospital” might be a typo. “Bob” is short for “Robert.” Maybe they are the same person.  In reality, it’s hard to definitively state “yes” or “no.” Instead, I had access to tons of information about each researcher, including email addresses, phone numbers, birth dates, and so on. I would assign scores (how reliable) and weights (how important) to each piece of data and if two researchers crossed a certain threshold, they were probably the same researcher and we didn’t have to spend valuable time and money trying to manually verify this.  This same approach should be taken to the definition of life. Instead of life being a binary “yes or no,” we should describe its “biopresentation,”   or “how present are the signs of life in a ‘thing.'” So for a virus:         Property  Score  Weight  Value      Metabolism  1  5  3    Homeostasis  0  4  0    Growth  0  2  0    Reproduction  3  4  12    Communication  1  5  5     In the above, a virus might have a biopresentation of 20. A human, however, would be radically different:         Property  Score  Weight  Value      Metabolism  5  5  25    Homeostasis  5  4  20    Growth  5  2  10    Reproduction  5  4  20    Communication  5  5  25     A human might have a biopresentation score of 100.    This would be particularly useful in describing abiogenesis. One hypothesis of abiogenesis is  :   Prebiotic atmosphere with ammonia, carbon dioxide and water vapor.  Lightning produces simple organic compounds that fall into solution in shallow water.  The compounds react further in a prebiotic broth, forming amino acids.  The amino acids link up with peptide bonds to form polypeptide chain proteins.  The proteins combine into more complex molecules that can replicate and metabolize simple substances.  Complex molecules and organic compounds form lipid membranes around themselves and start acting like living cells.   Traditionally, at step six is when we think “life”, but for biopresentation, we might starting thinking “lifelike” at step five. Maybe even four. Instead of life being a fixed point, it’s a gradual process. We don’t point to a thing and say “this is alive” because that gets us back to the arguments of what life is or isn’t. We simply describe its biopresentation.  This would be useful for biology today, too. Consider the endless arguments about whether or not viruses are live  . It’s often argued that viruses aren’t alive because they can’t metabolize outside of their host. That reduces metabolism to the primary (or sole) attribute of life.   But metabolism is the global term for anabolism and catabolism  . Catobolism is just breaking complex stuff into simpler stuff while anabolism is using that simpler stuff to do more complex stuff. Fire catabolizes, but doesn’t anabolize. Viruses do both. So maybe they’re alive? But some still reject that because, again, they can’t do that without a suitable host.  But what’s a host? Obligate parasites such as the protozoan Plasmodium   can’t survive outside of their hosts. Or consider humans. If I drop you into the middle of interstellar space without a space suit, you’ll stop metabolizing, too. So the context in which we can biopresent is also a spectrum. Viruses are starting to look more alive.  But wait! There’s more!  Most living beings have some ability to communicate with one another. Maybe all do. As it turns out, we’ve know for a few years that viruses can do this, too  . The human communication score might be at a “ten,” while viral communication might be a “one,” but fire’s communication is “zero.” The weight of this factor is probably pretty high, so fire’s biopresentation would be pretty low, but viruses could be easily crossing the threshold.  Pragmatist that I am, I want to skip the philosophical debates about whether or not something is alive and go with the pragmatic discussion of how closely something resembles life. However, we can’t ignore the philosophical implications.  Objections to Biopresentation  At this point, I can see many biologists howling at the above. In a 2013 Wired article entitled It’s (Almost) Alive! Scientists Create a Near-Living Crystal  , the lede reads:   Three billion years after inanimate chemistry first became animate life, a newly synthesized laboratory compound is behaving in uncannily lifelike ways. The particles aren’t truly alive — but they’re not far off, either.   The crystals created had metabolism and mobility, but couldn’t reproduce. You can read the original study at The Royal Society website  .  For these crystals, one could argue that a), they’re not alive, or b), what their relative biopresentation level is. Some people would be uncomfortable describing crystals as alive, but if we’re to search for life elsewhere in the universe, are we going to argue that life which doesn’t share the same biochemical basis as our own isn’t alive?  Or consider self-replicating Von Neumann machines  . By the biopresentation argument, they would likely score very high, but are machines “alive”? Or apply biopresentation to The Gaia Hypothesis  . Is our planet alive? That’s definitely a philosophical question, but in light of biopresentation, depending on what factors to include, we could make a strong argument that it is.  Why is rethinking life important?  If we go with a binary “yes\/no” for life, it means that we have defined a limited, possibly incorrect, set of properties necessary to define life. Something not meeting those conditions might be disregarded. If, however, biopresentation is adopted, we can start defining “lifelike” properties and perhaps not be so quick to dismiss something that is otherwise alive.    Is Biology the Continuum’s End?  If biology stems from chemistry stems from physics, is that the entire continuum? I don’t think it is. I think it’s pretty hard to describe viruses as being very sentient (the ability to feel and perceive things), but fish are. By the same token, it might be hard to describe fish as being particularly sapient (the ability to learn, self-reflect, and so on), though again, both sentience and sapience are probably continuums, as are most things complex.  So clearly biology can lead to sentience and sapience, but do they require biology? Possibly not. Consider the recent case of Blake Lemoine who contended that some of Google’s software had become self-aware  . While I tend to agree with Google that Lemoine was wrong, I am not so sure how long I can hold that position in the face of greater AI advancements. Since this is just electrons flying around silicon, can we argue that sapience doesn’t require biology? If so, the continuum could be across a tree or graph instead of a line. Or since humans built this, does that make the argument moot? I don’t think this is possible to answer, but it raises other interesting questions.  If we have a complicated chemical reaction that biopresents as zero (“pure chemistry”), could sentience or sapience emerge from that? Could sapience emerge from pure physics? This gets into the metaphysical realm and we cannot answer that. It starts teasing out hints of a god or gods. While I am an atheist, many of my friends are not and these question approach the realm of the ineffable, a term frequently applied to God. I worry that I sound like the atheist equivalent of Reverend Teilhard de Chardin  , a Catholic priest who tried to unify science and religion.  ","title":"Redefining Life as a Continuum","url":"\/blog\/redefining-life-as-a-continuum.html"},{"body":"    Introduction  The Drake Equation  (Not) Detecting Life  Assembly Theory  What If We Find Life?     Introduction  Few questions intrigue us as much as “are we alone in the universe.” Like many, I’m obsessed with the topic. So obsessed, in fact, that the National Library of Finland released an hour-long talk I gave on the subject.     I’ve also previously written on this topic and the above talk is an in-depth expansion of what I’ve written earlier, but hardly scratches the surface. But to understand a bit more about whether or not there is life elsewhere in the universe, let’s look at some of the background.  The Drake Equation  The Drake Equation is a famous probablistic equation created by Dr. Frank Drake in 1961. In this equation $N$ is the estimated number of intelligent civilizations we might be able to detect.  $$N = R_* \\times f_p \\times n_e \\times f_l \\times f_i \\times f_c \\times L$$   $R_*$ – Average rate of annual star formation in the Milky Way galaxy  $f_p$ – Fraction of those with planets  $n_e$ – Fraction of those planets which can support life  $f_l$ – Fraction of those which develop life  $f_i$ – Fraction of those which develop intelligent life  $f_c$ – Fraction of those generating detectable signals  $L$ – Lifespan of those signals   The variables $R_{*}$ through $f_{c}$, when multiplied together, tell us how many intelligent civilizations “mature” per year, to the point where they can be detected. Multiply that by $L$, the last term, and you have an estimate of the total number of intelligent civilizations we might detect at any one time.  At the time the equation was created, we didn’t know any of those numbers, but Drake himself estimated that $N$ was around 10,000  . Today, we have been slowly knocking those numbers down, one by one. Of those we think we know the what the values are, we have:   $R_*$ – Average rate of annual star formation in the Milky Way galaxy (6-7)  $f_p$ – Fraction of those with planets (50% to 100%; probably closer to 100%)  $n_e$ – Fraction of those planets which can support life (1?)   The last number, “Fraction of planets which can support life”, is often reported as $1$, but we already know that many moons in our solar system might support life . So we could easily have multiple “planets” per system which could support life. We think there are between 100 to 400 billion stars in our galaxy  , so that could easily lead to over a trillion heavenly bodies which could support life in our galaxy alone!  We don’t know how many stars are in our universe, but current estimate suggest there might be between $10^{22}$ to $10^{24}$ stars  . That’s up to 1,000,000,000,000,000,000,000,000 stars!  So what about the number of planets which develop life, have intelligent life, and so on? We have no idea what those numbers are. However, that doesn’t mean we know nothing about those numbers. We exist, Therefore, all of those numbers are greater than zero. The first, $f_l$ (fraction of planets which develop life) is what we’ll focus on here because if that number is too low, none of the others matter.  (Not) Detecting Life  You may have read in the news lately about a new method of detecting life on Mars: DNA sequencing at the picogram level to investigate life on Mars and Earth  . This is remarkable work, but to point out the obvious—which the authors of the work are certainly aware of—what if the alien life doesn’t have DNA?   In a paper entitled Is There a Common Chemical Model for Life in the Universe?  , by Benner, Steven &amp; Ricardo, Alonso and Carrigan, Matthew. (2005), point out that most “universal” probes for life target the ribosome, present in virtually all life on Earth. Again, what if the life doesn’t have ribosomes? Using this as a tool to detect life won’t work and we know nothing about alien life (if it exists), so can’t make strong statements about what that life is likely to be.  There might be another approach, though. Two Viking landers arrived on Mars in the 1970s and their labeled release experiment might have detected life . In short, in separate experiments conduction over 6,000 kilometers apart, the landers injected Martian regolith (the surface material on Mars) with a radioactive nutrient solution and something interacted with the solution and released the radioactive carbon-14. It looked an awful lot like metabolism. It might just have been a chemical reaction. We don’t know.  What’s interesting about the Viking Lander experiments is that they weren’t searching for specific chemicals. They were looking for a biosignature indicative of life. Yes, we made assumptions about the “nutrient” solution, using seven molecules found in the Miller-Urey experiment  , but trying to have an agnostic approach to detecting life was promising. That brings us to Assembly Theory.  Assembly Theory  Detecting alien life is a tricky business because:   We don’t know if it exists  If it does exist, we have no idea what it’s composed of   Assembly Theory was first proposed by Professor Leroy Cronin  , the Regius Chair of Chemistry in the School of Chemistry at the University of Glasgow. Instead of just asking what a molecule is , he asked how it came to be. In other words, how was it assembled? He’s established the notion of a “molecular assembly index” (MA) which represents the minimum number of steps necessary to assemble a given molecule.    Consider the word “going.” Imagine that each letter is an atom floating around which can freely attach to other atoms to form words. The MA of a single letter is zero because it already exists. However, when you add the letter “o” to the letter “g”—forming “go”—you have an MA of 1, because one step has occurred. When you then add the letter “i”—forming “goi”—you have an MA of 2, and so on. The word “going” therefore has an MA of 4. Of course, by the constraints we’ve listed, all other possible combinations, such as “oin” (MA of 2), “in” (MA of 1), and so on, are floating around in our primordial soup.  But consider the word “hodgepodge.” By the time we’ve assembled “hodgep”, we have an MA of 5. You might think the word “hopgepodge” has an MA of 9, but remember that substrings of “hodgepodge” are floating around, too. One of those substrings is “odge” and can attach to “hodgep” in a single step, giving us a final MA of 6, not 9.  Of course, chemistry is much harder than spelling. Molecules are 3D structures and individual atoms don’t form bonds randomly with all other atoms. But you get the basic idea.  In the paper, Identifying molecules as biosignatures with assembly theory and mass spectrometry  , (Marshall, S.M., Mathis, C., Carrick, E. et al.), Professor Cronin and his colleagues took the time to:   calculate the complexity of several million molecules and validate that their complexity can be experimentally determined by mass spectrometry.   Their findings?   These results demonstrate that we can identify the living systems by looking for mixtures with an MA greater than a certain threshold. In the case of the analysis presented here it appears that only living samples produced MA measurement above ~15.   In short, molecules with an MA above 15 were invariably produced by life, while molecules below that might be produced by both abiotic or biotic processes. If these results hold, we might have a “universal” method of detecting life, with the caveat that if there’s life that only generates molecules with lower MA numbers, we’ll still not detect it. However, this approach seems rather promising.  To give some context, sometimes you’ll read breathless news articles proclaiming that we’ve found “complex” organic molecules in space. For context, organic molecules are just molecules with carbon-hydrogen or carbon-carbon bonds They’re all over the place because carbon is one of the easiest molecules to form bonds with. One of those “complex” organic molecules found in space is ethyl formate. It has an MA of 3. The molecule n-propyl cyanide has also been seen in space. It also has an MA of 3.  In fact, most complex organic molecules in space have rather low MA numbers and it’s not hard to imagine abiotic processes producing them. On the other hand, we may have detected tryptophan around the IC348 star system  , about 1,000 light years away. Tryptophan has an MA of 11.  (If you’d like to check these numbers, I wrote a script to do that for you  , but it requires the Molecular Assembly REST API  . That doesn’t always seem to return results. In correspondence with Dr. Cronin, he confirms they’re working on a new version)  Throughout all of the above, keep in mind that Assembly Theory, like all new scientific endeavors, has its detractors.  What If We Find Life?  Three things. Given that there are possibly a trillion or more candidate places for life to come into being in just our galaxy, given that we know $f_l$ is greater than zero, and given that we’re seeing more and more complex chemistry in our own solar system which would be easy to explain biologically (were we to know the biology is there), I’m gradually leaning to the idea that we are probably not alone in the universe. Of course, that’s a hunch, not science.  In the fifth century BCE, the Greek philosopher-scientist Anaxagorus speculated that there was life on the moon  . Many have speculated about this throughout history and certainly in pre-history. That we have so many religions asserting non-human intelligences with whom we can communicate is part of this longing of ours to know that we’re not alone in the universe.  Finding life, but not intelligent life, will not be enough to satisfy us, but it will be an amazing step forwards in filling in the last of the variables in the Drake Equation. Further, if we find life in our own solar system, but it has a different origin from our own, it’s virtually impossible that the only two abiogenesis events in the universe happened in the same solar system. The universe would be teeming with life.  For those wondering about abiogenesis (the origin of life from non-life), here’s a video by Sabine Hossenfelder describing our current thinking on the topic.     ","title":"Searching for Alien Life","url":"\/blog\/searching-for-alien-life.html"},{"body":"  I wrote this a while ago, exploring first-person, present-tense, non-linear story telling. That wasn’t the intent, but when the story came to me, it was there and it wouldn’t go away. I don’t say it’s a great piece, but I’m proud of it. I also find that first-person, present-tense makes it easier for me to avoid filter words  . I have no idea why. I have a chapter from my novel that’s written in first-person, present tense, and it still gives me chills to read it. Narcissism is fun!  It’s worth noting that the main character is based on a friend. We worked together and he told me what it was like growing up black, in a gang, in Portland, Oregon. I’ve taken a few liberties, but his voice rings strongly in my mind and I think I’ve been faithful to it, though I’ve toned it down. A white author writing about black experience cannot use the n-word.  The \" now \" and \" then \" are recent additions by me because some early readers complained it was too difficult to follow the non-linear story line. However, I can’t simply re-arrange the story in a linear fashion because that destroys it. Hence, my added cues for the reader. If they don’t help or distract, let me know in the comments. In fact, I’d be grateful for any feedback you care to leave. I want to be a better writer.  Also, I know it’s the US Marshals who handle the witness protection program. I wrote “FBI” because I wasn’t sure most people would know who the US Marshals are.  The Names We Never Say  Now.  “Parents died in a house fire when I was a kid, so no, I don’t have no childhood photos.”  That shuts ‘em up. It always does. I roll up my left sleeve. A mess of white scar tissue on black skin.  “That’s my childhood photo.”  I got that from a night drinking and messing with a barbecue, but I didn’t tell them that. Heck, I shouldn’t even have shown them the scar, but lying always made me nervous and I felt like I had to back it up. Old, stupid habits die hard.  “James, I’m so sorry.” Frank’s squeeze—don’t even remember her name and there’s no point because she’ll be gone next week—goes beet red with embarrassment. I feel sorry for her. Frank’s my best friend now, but he’s a bit of a bastard about women. It was the only thing we argued about. I’ve found God now. I think. Not sure if he’s found me.  “No worries, girl. You didn’t know. Gonna get another beer and if you two ain’t drinkin' with me, I’m gonna be mighty offended.”  Middle of a Saturday afternoon and Frank’s here to watch the game with me. My wife and daughter are out on a mother\/daughter field trip and I have a day off to watch LeBron destroy Miami. It’s a hot summer day and a few Coronas with lime go down a treat. I’m in the kitchen, reaching in the refrigerator when I hear the clang of the mail slot in the front door.  I walk back into the living room and there’s Frank, reading an envelope.  “Who’s Jamal?”  With that innocent question, my life is over for the third time.   Then.  You have no idea what it’s like growing up near Martin Luther King Avenue in Portland, Oregon. I had the luxury of traveling a lot before I got busted the second time and every place that has an “MLK” Boulevard\/Avenue\/Promenade—nothing ever as pedestrian as “Street”—is always through a poor black neighborhood and that’s where I grew up. That’s where I dealt. That’s where I joined a gang. That’s where I lost my virginity and, as a result, acquired my first scar. A stab in the gut courtesy of her boyfriend.  I’m not stupid. I didn’t brag to nobody, but one night hanging out, there he was. His name was Aaron and he walked up, smiled, and said “‘Sup, Jamal?”  I just nodded and he slid a knife in my gut like it was the most natural thing in the world. To him, I suppose it was. People were screaming and Aaron just walked away.  I told the cops I didn’t know who it was. You settle your own shit. Course they sent a black cop round. Like I’m gonna trust one of them. I went looking for Aaron later. I thought I sort of deserved what I got, but you gotta save face, right? Everybody said Aaron disappeared. No idea where he went off to.   Then.  I didn’t plan to grow up dealing; I wanted to go to college and study literature. Momma didn’t have much, but she loved books. Naturally, I loved books too, but lettin' on that I huddled under the blanket, switching from Heinlein to Faulkner to “Poets of the Nuyorican Café” made me a joke. So publicly, I read biographies of Malcom X, Nelson Mandela, or anyone else that wasn’t going to piss off my friends too much, though I liked them, too.  But nobody hired boys named “Jamal.” At least not good jobs. At least not in the whitest major city in the good ol' US of A, love her or leave her. Which is how I got busted the first time, dealing while working at a car wash. I wasn’t stupid and I’ve got a feel for people, but it’s not perfect. The undercovers dropped a name and asked if they could score. Later they lied their lilly-white asses off in court. There was no entrapment; I offered it to them. Guilty.  But I never grassed on no one. You don’t get two years for a first offense, but with a name like “Jamal” and refusing to cooperate, I had a DA in a bad mood. Got out six months early for good behavior, hit the streets and life was over.  I tried to go straight. I really did. But when you have to explain time in prison, you don’t get no real jobs, which is why I felt real lucky to have Christine. She was Aaron’s ex, the one I slept with. He left her a broken nose as a parting gift, but she was lucky and it healed sort of upturned. It was cute, but I hated Aaron. You don’t hit women. No way.  Christine was a receptionist at a car dealership and got me a job as a salesman. You piss clean in a cup and talk like nobody’s business and they don’t care about your past. They just want to know if you can make them money.  Christine was special. For the first time in my life, I felt like things were going good. I didn’t feel too comfortable round her mom, though. She didn’t care about my past—too many good boys get caught up in bad things—but she creeped me out. I was sitting there, sippin' iced tea, waiting for Christine and talking to Christine’s mom, Gisele. “Gee-seh-luh.” What sort of name is that?  You didn’t lie to Gisele. I learned that early on. I never felt good about lyin'. I could lie when I had to, but not with Gisele. She was a human fuckin' lie detector. I once told her I sold 18 cars last month. I just wanted her to know I could take care of Christine. That’s exactly what I was thinking. Gisele just looked at me and said “I know you’ll take care of Christine, but you didn’t sell no 18 cars last month, Jamal.”  She could have found out how many cars I sold, but how the hell did she know what I was thinking? I never lied to her again.  So there we were, drinking iced tea, and I said Clinton was going to crush Trump in the election. Gisele looked sad and said, “no, she’s not.”  And just like that, I knew Clinton would lose. Gisele did that to you.  When Christine got home that night from college, she stepped into the living room and plunked herself down on the old, ratty sofa and curled up next to me. I wrapped my arms around Christine and knew with dead certainty that I loved her. Probably the only reason Gisele let me in the house. She knew.  And then she didn’t let me in no more. I wasn’t a great salesman, but the other guys knew my past and kept asking if I could hook ‘em up. That’s how I got back in. I wanted to make money for Christine and here it was. Just one look at her big, brown eyes and I’d have given her the world if I could. So I lied and said I was making good commission on my cars, when really I was hooking up the other salesmen. And then I started hooking up more than other salesman. That’s how it is.   Now.  “Jamal? He’s my cousin in San Francisco. Don’t know why I’d be getting mail from him.”  Frank hands me the envelope and I look at the address. “Jamal, c\/o James Hillman.” I’m still Jamal inside, but Frank and everybody else knows I’m James. Except for my wife, Angela, née Christine. Even our daughter, Sarah, don’t know our real names. I hated lying to her about it, but I love her as much as Christine. No way in hell I’m going to give her a gun to take her parents away from her.  I put the envelope aside and Frank forgets about it. Every nerve in my body is screaming, but I wander back to the TV, acting like nothing was wrong. I still don’t know who won that game.   Then.  New apartment. New king size bed. Mind wandering.  “What sort of name is Gisele? I never heard no name like that.”  Christine was lying in my arms, our bodies still covered with sweat, and whispered to me, “her name’s not Gisele.”  “What the hell does that mean?”  “She has this thing about names and won’t tell anyone her real one. Says it gives people power over you. And stop talking that way, you’re better than that.”  “So what’s her name?”  Christine sat up and looked down on me.  “No, Jamal,” she said. “She maybe has weird ideas, but she’s my mother. I won’t dishonor her.”  Christine was sort of a Christian and honoring her mother and father (God rest his soul) was important to her. I say “sort of a Christian” because she knew I was making more money than selling cars could provide, but she never asked. I liked to think it’s because she loved me, but I never asked, either.  I sat up and wrapped my arms around Christine. I knew not to press about Gisele’s name and hugged Christine tightly, but this bothered me. Everything about Gisele gave me the willies and at this point in my life, anything I didn’t understand was a threat. Except I didn’t understand Gisele and she didn’t feel like no threat.   Then.  A couple of years before that, I got some really bad good news. I was a small-time dealer, now selling cars and coke and my supplier invited me to dinner on a night Christine was in school. I went round to the restaurant, but my supplier wasn’t there. Lamonte was. Lamonte was my supplier when I went to prison. I never dropped his name, but maybe he didn’t know that. But when he saw me, he smiled, and waved me over. “Sit down, Jamal. It’s time to talk about your future.”  I looked round, but no one in the restaurant said “cop” to me. Not even Lamonte. A waiter walked up and smiled, so I sat. I ordered a beer, took a menu, and waited for the bad news.  “I hear you play straight. And I know you do. You could have sent me up, but you went to jail rather than drop my name.”  “Prison. Jail may be shorter, but at least they got shit to do in prison.”  Lamonte nodded; he knew what I meant. You don’t, but he does. In jail you sit on your ass and play cards, waiting a few months to get out. You’re bored stiff. In prison, you’re there for a long time, so they have weight rooms, college classes, libraries, you name it. Jail doesn’t have shit, but in prison, they give you a chance to better yourself. Maybe you’ll come out with a degree. Or maybe it’s a crime finishing school, with professors who were stupid enough to get caught.  The waiter came by and I ordered. When he left, Lamonte leaned forward and said “You had my back when you didn’t have to. Now I got yours. You say the word and you don’t have to sell no more god-damned cars. I’m doing good and I want people I can trust. I trust you.”  I sat back, scared as hell. Christine looked the other way about the extra cash I was earning, but that’s only because I had a job. She was the one good thing in my life and I didn’t want to fuck it up. That’s what I told Lamonte.  “You’re a good man, Jamal. I appreciate what a good woman can do for a man, but what can you do for her? You’re a felon. You know you’re never going to get a real job. Oh, you might be a sales manager some day, earn whatever small bit of green the man wants to toss your way, but you really want to take care of her? I can offer you that. It’s a real job. I start you at five grand a month and you run some dealers. I won’t lie to you. I ain’t promisin' nothin' bad’ll ever happen, but if it does, we got each other’s backs. And you treat us right, we treat you right. You can keep Christine in college and have nice things.”  And that’s how life ended the second time, but at least it paid well.   Now.  It’s two days before I open that envelope. I know it’s stupid, but I’m scared. I don’t know what the hell to do. Christine—excuse me—Angela probably knows something is up but I tell her it’s just work. With the economy the way it is, she accepts that. I hate to lie to her, but I’m really thinking about Sarah. When she was born, she brought joy to our lives like we had never experienced. Oh, we were as happy as our circumstances could allow, but this felt like it was for real this time. Past sins were absolved and beautiful Sarah laughed and cried and hiccoughed and I even started going to church. My life was almost complete. Even changing diapers had made me smile.  Sarah had never known any life but what we had. I didn’t want to take her out of school, but there’s an envelope in my hand. Hell, I like going to PTA meetings. I want Sarah to have everything I didn’t. And so did Angela. We have the life we always wanted. It isn’t perfect, but it’s stable. And always with that tinge of fear. Now I know I should call the FBI and I know they will take away our life and give us a new one. Sarah will cry for her friends in school and I have no idea what to say to her.  I don’t know what to expect when I open that envelope, but I certainly don’t expect what I find.  I unfold the paper and read. Or at least I thought I would. The paper is blank. I turn it over, but nothing is there. Paranoid, I look in the envelope for weird powder or whatever the hell anthrax looks like, but I see nothing. I sniff the air. Nothing. I sit. I wait.  I have tears running down my face when I notice the writing on the back of the envelope.  “Gisele.”  And then the mail slot clanged. There was another envelope on the floor.   Then.  Life working for Lamonte was good. Lamonte was careful and if a dealer started using, they were out. You couldn’t trust them. I was one of several “account managers” and we were a tight group. We knew Lamonte’s star was rising and we along with it. Because I talked proper when I needed to and didn’t mind wearing a suit, I soon found myself working directly with several favored customers. You’d probably even know the names of a couple of them, but you won’t hear them from me. When I landed a narc as a customer, we had a nice celebration. Portland cops had worked real hard to clean up, so getting inside news was a real score. He got his shit for free, sometimes wrapped in Benjamins, and we heard rumors nobody else heard.  That’s how, when Lamonte got busted, I had his job. Lamonte was true to his word and didn’t name names. And Christine didn’t ask questions about my import\/export business and cried when she thought I wasn’t looking. I was a “district manager” and the account managers reported directly to me. I didn’t change nothin'. What worked for Lamonte worked for me.   Now.  I open the envelope right away this time. A single sheet of paper. A single word. “Aaron.” Written on the back of the envelope was “Gisele,” in Gisele’s handwriting.  Of course it wasn’t Gisele. She died years ago.  I’m staring at the envelope and I dimly remember that like Angela and myself, she had a different name. How many people knew that? Sure, she had different reasons, but still. She was buried years ago and now I have this envelope in my hand and a sheet of paper which reads “Aaron.”   Then.  I met Aaron again shortly before my second bust. I was pulling down a cool seven grand a month. I never skimmed, I didn’t cut the dope too much and when I fired a dealer for using, I gave them severance pay out of the expense account. Then I told ‘em to get lost.  One day on an exchange, Big Boss told me to carry. We were dealing with some new boys. They were unknown, but they had lots to move. Normally I don’t hold no truck with guns, but when the big boss trusts you to watch his back and you’re looking to move up, you do what you’re told. So I carried. And there, in an old warehouse, was Aaron providing muscle for the new boys.  I slowed down. I guess the big boss sensed something was up and he slowed down, too. Then they slowed down. All of a sudden, a clean exchange became real tense. I could see Aaron’s eyes moving from man to man, doing the math. I wanted to hurt Aaron bad. Not for what he did to me—I could forgive that—but for what he did to Christine. But I can do math too and the numbers didn’t add up. Aaron and I just stared at each other, hatred in our eyes, but the deal went down.  Big Boss later asked me what happened but I told him I just had a bad vibe. I stuck to the story. No sense worrying the man. Aaron must have lied to his big boss too because we had more deals and I always carried. You don’t need to tell me twice.  When Aaron finally came looking for me, he shot the big boss and I killed Aaron. Or at least I tried to. Lucky shot got him in the leg. The Feds offered a deal: it was self defense if I testified, Christine and I would get the Witness Protection Program, provided I kept myself clean. If I didn’t testify, I’d go down.  I don’t grass, but I had Christine to think about. It was a hard decision. I tried to keep my account managers out of it, but no deal. And I had to point out the big bosses. Mine survived being shot and Aaron’s boss went down, too. One stupid fool and everybody’s looking at me. Well, everybody except Aaron. The slippery son of a bitch was offered the deal, too. He took down his side, I took down mine.  So Christine and I live in hiding. New ID cards. New social security numbers. I’m a district manager of a snack vending company—yeah, I get the irony—and Christine, bless her, didn’t leave my side. She was just happy I was out.  And then we had Sarah.   Now.  You’d be surprised how many people die in the Witness Protection Program. The FBI drilled this into us repeatedly. All it takes is one moment of homesickness, one quick phone call, and if the people who want you want you bad enough, you’re dead. One fool went back to his old home and turned the doorknob and it blew up.  Christine and I knew the score. Neither of us had anything to go back to. Christine became Angela and I became James. We were moved to a new city in another state. The FBI got me a job, a small house, and a phone number for emergency use only. No friends. No social circle. No way back. Pretty fuckin' shitty deal, but it beats dying.  But we had each other. That’s enough.   Then.  Gisele died unexpectedly. She was in great health and whenever Christine wasn’t with me or at school, she was with her mom. So when one day Gisele didn’t answer her door, Christine panicked. She just knew something was wrong. Shortly after, we were at Gisele’s funeral. Brain hemorrhage. The doctor promised us it was painless and looking at her in the casket, she looked peaceful, but then, I guess that’s the mortician’s job.   Now.  I’m looking at the sheet of paper and know I have to go to the FBI now, but then there’s Sarah. What do I do about her? She’s not old enough to know what’s going on and there’s no way she’ll understand if we call her a different name. She won’t know why she has to leave her friends at school and Angela’s going to be terrified. So I do what I have to do. I buy a gun.  Normally, ex-cons don’t get to buy guns but James Hillman is not an ex-con. I buy a shotgun to avoid the waiting period and I go home and head down to the basement. I turn on the light bulb and ignore the damp, musty smell. I open my toolbox and a few strokes of a hacksaw removes most of the barrel. A bit of work with some files and cutting down the stock and I have a nasty short-range weapon. A nasty short-range weapon that’s pretty easy to hide. Pretty illegal, too, even if James Hillman isn’t an ex-con. I’m going to have some trouble explaining this when I get caught.  I will get caught. I know that, but I don’t care. But Christine and Sarah will be safe. I really have no choice.  I think about this a lot and I figure that if Aaron’s after me, he’s a lone nut. Otherwise, Gisele would have said something different, right? But no, Gisele’s dead. Who the hell sent that damned envelope?  Aaron’s Lee Harvey Oswald and I’m John F-in' Kennedy. Except JFK wasn’t married to Oswald’s ex. JFK didn’t know Oswald was coming to seek revenge for JFK being with Jackie. And I will kill Aaron and the cops will take me away and Christine and Sarah will be safe. That’s all I care about.   Now.  I stop staying late in the office and bring my paperwork home with me. Christine—or is she Angela?—asks what’s going on and I tell her I just want to spend more time with her, even if I have to work. I thought I caught her crying once. She knows something is wrong just as she always does.  Sarah likes to crawl into my lap. I kiss her forehead and keep working, my mind trying not to think of the sawed-off shotgun in my briefcase. On a hot Saturday, I’m not surprised when the third envelope falls through the slot. They always arrive when Angela\/Christine and Sarah\/Sarah are out.  I open it and read two words, “Tomorrow Night.” On the back of the envelope was a single name, “Lizbeth.”  That night, curled up next to Christine\/Angela, I ask her a simple question.  “Her name was Lizbeth”, Christine murmurs.  But by now, I already knew that.   Now.  Sunday. Sarah’s best friend was always asking her to sleep over and, for the first time, I said it was a good idea. I am over-protective of Sarah. Christine thought Sarah staying with a friend was wonderful and had already started to plan a romantic evening but an emergency phone call from work took me away.  And an emergency phone call from Frank took Angela away, too. He says he’s broken up with his girlfriend and is upset, but since I’m not there, will Christine come? I told him to keep her away the entire night, whatever it took. I’ll be happy if she’s mad. Dead people don’t get mad. As she drives off, I huddle down further in the bushes, the shotgun in one hand and a cell phone in my pocket.  Aaron shows up just before midnight. The lights are off and as he limps to the front of the house, I watch him, quiet as the names we never repeat any more. He walks up to the front door and feels under the mat for a key. That’s good because I left one for him. If he breaks in, maybe I can get away with shooting him in the back. As he carefully turns the key in the knob, I raise the shotgun.  And then I put it down again. Good people might do bad things, but I couldn’t do this. Aaron tiptoes inside and I lift the cell phone and call the special number the FBI gave me for situations just like this. And that’s when Christine pulls back into the driveway. And that’s when Aaron steps outside. And that’s when Aaron pulls a gun and points it at Angela.  Or is she Christine? I’m confused. Gisele’s in my mind telling me I’m a good man who does bad things. I’m not. I’m a bad man trying to do good things.  I shoot Aaron in the back, rack, shoot him again, rack, shoot him again and keep going until the shotgun is empty. Christine is screaming.   Now.  You might think I’d get in trouble with that shotgun, but when you’re in the Witness Protection Program, some problems go away. “Brave Homeowner Kills Intruder Threatening Wife” read the headline. Any publicity usually means you need to leave. Publicity and knowing someone has found you guarantees it. I don’t know what we’ll tell Sarah, but the FBI tells me they already have a plan for that.  When we move into our new house, there’s an envelope lying on the floor in the foyer. There’s no name on it, but it has a single piece of paper with two words, “Thank you.”  ","title":"Short Story: The Names We Never Say","url":"\/blog\/short-story-names.html"},{"body":"  Introduction  I’m an American who’s lived in the US, the UK, Japan, the Netherlands, and now France. People kept emailing me over the years to ask me how to move to a foreign country. As it turns out, the answer is complex and I got tired of cutting-n-pasting the same email over and over again. So I started a blog entitled “Overseas Exile.” I posted several times a week for many years, but I let the domain lapse and now someone else has snapped it up. You can still read my old posts  , but I felt, given the current climate in the US, that it’s worth sharing some of this here.  This wasn’t my first post, but it was one of the most important, despite the silly premise.  Count von Europe  So your eccentric great-aunt Gertrude has invited you over for dinner and when you arrive, you find a mysterious stranger with her. She introduces him as “Count von Europe”. After a long and pleasant conversation over food and drinks, the Count says “your aunt invited you over because I need someone to watch my Bavarian castle while I’m away for a year. I’ll also pay you €50,000 a year and let you borrow the Bentley when you want to travel around Europe on your days off. When can you fly to Germany?”  Sadly, for many people I’ve spoken with, their “dream” of living abroad is little more than “Count von Europe.” I see people on message boards saying things like “I want to move to Italy. Anybody got a job\/marriage\/house for me?” Honestly, it’s not going to be that easy. However, you can do it if you plan things right and understand what’s involved. When I talk to people about this, I sometimes use “Count von Europe” as a thought experiment, but the ending is a little different. He makes the same offer, but with a catch: you have to leave tomorrow.  First, we’ll have a slight digression. Many years ago I used to sell cars (don’t hate me for that. There are plenty of other things you can hate me for). One thing management drilled into our heads over and over again was that 80% of people only agree to drive home in a car after being asked to buy five times.That’s because people have objections. “The price is too high.” “I wanted red, not blue.” “I’m just looking.” My job as a salesperson was to understand and overcome all objections the customer had. If I could do that, they’d agree to buy the car, but it’s harder than it sounds.  Objections  If you want to take up Count von Europe’s offer, overcome your own objections. If you can do that, you’re one step closer to moving to another country, so let’s look at those objections.  I don’t have a passport.  This one boggles my mind because I’ve been surprised at how many people say they want to live in another country but don’t have a passport. If you’re from the US, go here to apply for a passport  . Otherwise, it’s fairly easy to find out where to apply.  I’m locked into a long-term lease.  Then find a way to get out of that lease. When I was offered a job in another country in 2001 I had three months left on my lease. I went to the landlord and explained the problem. She was actually very gracious about it and let me end the lease early without penalty. It never hurts to ask!  Failing that, convert your lease to a monthly lease when it ends. Or have enough cash on hand to buy out the lease. Or when it ends, become someone’s roommate.  But I own the property I live in!  Then sell it. Or rent it out and rent a room somewhere. Or find a property management company who will rent it out for you.  I have a wife\/husband\/partner\/children, etc.  That’s a tough one. Right now you have to decide what you want to do. If you have obligations to others who don’t want to leave (I’ve been there and it’s tough), then you’re stuck. I can’t\/won’t offer advice here other than to suggest taking them on a vacation to the target country. It’s a lot easier to appreciate something if you know something about it.  I have a cat\/dog\/iguana.  You can still move overseas, but you have to make sure you conform to your target country’s regulations for shipping your pet there. The “Transitions Abroad” Web site has a good article to help you understand the basics of moving a pet overseas  .  Other Objections  If you get a chance to leave tomorrow, what’s stopping you? Figure out what that is and if it’s more important than living abroad. And fix it.  Conclusion  Think about these objections and any others you may have. This is the starting point for being able to move overseas. If you find that you could say “yes” to Count von Europe, you’ve gotten enough of your life in order that you can make this happen. This is key to realizing the dream of moving abroad. I don’t expect that you’ll actually be able to leave at the drop of a hat, but if you can, it’s much easier. Think how much cheaper and faster it is to move to a foreign country with a backpack as compared to an entire household of goods.  ","title":"Why You Will Say \"No\" to Living Abroad","url":"\/blog\/why-you-will-say-no-to-living-abroad.html"},{"body":"    My First Weddings  Why I Stopped Officiating Weddings  Why I Started Officiating Weddings Again  The BBC     My First Weddings  Note: Much of what follows is from memory, but the ‘seventeen weddings' described near the end are from local copies of old blog posts I had written on the topic, many years ago. Hence, the detail.  I have, to date, been an official participant in 21 weddings in the US since I turned 18.   Twice I was the groom. The other 19 times, I was the minister. The story gets weirder from there.  My first wedding was three decades ago. It lasted five years and she and I grew apart. The divorce was difficult, but then, they often are. I hope she’s doing well.  My twenty-first, and last, wedding was in Tower Bridge, London, on June 20th, 2010, to my lovely wife Leïla. We’re still happily married, with a beautiful daughter, Lilly-Rose.    Our wedding. Source    Being a husband and father has been exhilarating, wonderful, and sometimes exhausting, but I wouldn’t trade my wife and daughter for anything.  Getting back to the wedding, the date, June 20th, was selected because it’s my birthday. I have no excuses if I forget. She knows me too well and I can’t fault her for that.  Curiously, this is my second wedding on my birthday. The first “birthday” wedding was also the first wedding I officiated at. My friends, we’ll call them Alice and Bob, called me, frantic because they lost the judge who was going to perform their wedding at a lake. Apparently he was retired, senile, and had no recollection that he was to perform their wedding.  I, however, was neither old nor senile (I’ll not swear to either today), but I am an ordained minister. Despite being an atheist, I was ordained by The Universal Church  in September of 1990. Back then, you had to do it the hard way by sending them a postcard with your details. Today, you can do this online.  Because I was an ordained minister, Alice and Bob explained the problem and asked if I could officiate at their wedding. I told them I would call them back after I found out. I called my church and they agreed to Fedex™ the paperwork to me, free of charge. I called my friends and told them I’d be honored to preside at their wedding. I neglected to tell them it was my birthday; that would have been bad form.  I filed my paperwork with Multnomah county, in the US state of Oregon and was quickly approved, though not without some odd looks. I was wearing a pair of black earrings and a silver\/grey shirt that buttoned up the side, just the sort of thing I’d wear for a Goth Night of clubbing. But Portland being Portland, odd looks were all that came of that.    The first wedding I officiated at.   The wedding at the lake went off without a hitch and I got a good laugh when I pointed out that the witnesses were named “Tom and Jerry.” I also discovered, as a minister officiating at a wedding, it’s easier to pick up women then you would think. The date went poorly and my friends later told me that had they known who I met, they would have warned me not to. (Amongst other things, on our second date, not only did I find out that she was hopped up on little pink pills her former boyfriend had brought her from Mexico, she asked me to convert to Judaism because she had recently converted and couldn’t date outside of her faith. She was also an atheist. I declined a third date).  That couple also went their separate ways, so my first two weddings, one as groom and one as minister, ended in divorce.  Why I Stopped Officiating Weddings  The third wedding made me swear off officiating at weddings ever again. It was a favor for a friend of a friend. A couple couldn’t afford much, so they wanted a wedding in their home. I agreed, and spoke with the couple on the phone. I asked them about the theme (“non-religious”), who they’d like to include (“uh, we’re not sure”), if they’d be exchanging rings (“we don’t know”), and when they could send me a copy of their vows (“can you write them, please?“). Not one answer was particularly bad, but all of them together, coupled with very strange vibes from the phone call told me that something was, well, off.  I wrote their vows and called them back to arrange a rehearsal. They were busy, they said. Every time I called to try to arrange a rehearsal, they had an excuse. I simply could not get a rehearsal with them. So I finally said I’d show up to the wedding early, meet with them, and have a quick run through of the vows. In turn, I shortened the vows, knowing this was going poorly. It’s a very good thing that I did.  The day of the wedding, my friend (er, we’ll call him Charlie), who asked me if I would help the couple out, drove me to their place in the middle of nowhere. It was early October and they had decided on a Halloween-themed wedding. As we pulled up the house, little white trash bags hung from the trees. I later realized they were supposed to be ghosts, but they looked for all the world like Klan hoods.  As I walked inside, I immediately assumed a “minister” persona, wanting to put the people at ease. Charlie introduced me to sexy vampires, unshaved cowboys, and one under-age Pocahantas   who wore an ill-fitting costume and no bra. She was flashing everyone but no seemed bothered. I bit my tongue as I smiled. The phrase “feeling like a nun in a brothel” never felt more appropriate.  I was ushered into the back to meet the bride and while she stood their, half-naked, she read through the vows quickly and handed them back to me, assuring me they were fine. I then met the groom in a back bedroom and he told me he was busy and had no time to read the vows. I was brushed off quickly. It felt weird. No matter. I had shortened the vows because they wouldn’t meet for a rehearsal, but this was still very unsettling.  Finally the ceremony started, the happy couple faced me, their assembled family and friends faced me, I handed each of them a short copy of the vows they would read, and I started.  “Dearly beloved, we are gathered here today to celebrate the marriage of ...”  And that, dear reader, is when I realized that in the two months of failing to arrange a rehearsal, I had no fucking idea what their names were.  “... of the lovely bride and her handsome groom.”  The audience smiled. I had gotten away with it.  I then turned to the bride and asked her to read her section of the vows. I was proud of the non-religious, life-affirming, gender-equal vows I had written. None of this, “honor and obey” nonsense. She read the vows and the audience smiled again.  I then turned to the groom and asked him to read his section of the vows. He stumbled and I discreetly pointed to the section clearly labeled “groom” so he could recover.  That was when I discovered that the groom was functionally illiterate. He couldn’t read the vows. He slowly, awkwardly, sounded out the words and I held a supporting smile as I died inside. He was probably dying, too. As was his fiancée. This was a low-point in my life.  After the groom finished, I quickly got through the “do you take your bride ...” and “do you take your groom ...“—still omitting the names I didn’t know—and finished with, “you may now kiss your bride.”  After the kiss finished and the applause died down, I had them turn to face the audience and said, “I now introduce you to the happily married couple!”  More applause, followed by dead silence due to our lack of rehearsal. I had no idea what to do next. The audience stared at us. I stared back. I silently cursed Charlie (sorry, Charlie). Then I piped up with, “how about a question and answer session? I’m sure we’d all like to what their life plans are.”  That’s me, the professional speaker. Always prepared to improvise.  After another awkward pause, the audience got into the swing of things and started peppering the bride and groom with questions as I embarrassedly slunk away to the kitchen, desperately in need of a drink. That’s when I discovered it was a “bring your own beer” wedding and no one had bothered to tell me. I was filling a glass with water when some kind soul hurried up and pressed a can of room temperature Olympia beer into my hand. Olympia beer must always be served ice cold because you desperately want dead taste buds when you drink it, but I was grateful.  Later, Charlie and I stood on the back porch, smoking, drinking, and rather uncharitably speculating about how long the marriage was going to last, when the back door slammed open. An angel with a cigarette dangling from her lips burst out, cussing up a storm. Slipping back into “minister” mode, I asked, “what’s bothering you, child?”  “My sister married an asshole!”  I told Charlie I was never doing another wedding. He was, and still is, a friend, but I can never quite forgive that nightmare.  A couple of years later, I learned that the smoking angel was being charitable. The marriage ended very badly.  Why I Started Officiating Weddings Again  But that’s only four weddings. Given that I had sworn off weddings, it’s probably a surprise that I officiated at seventeen more. Over the course of two days.  In March of 2004, Multnomah County of Oregon briefly legalized same-sex weddings  , a full two months before Massachusetts started issuing marriage licenses to same-sex couples  . In fact, the political mess was so awful that at one point, Bend, Oregon banned all marriages  , whether they were same-sex or not.  The Multnomah County ruling was a contentious affair, with the city councilors passing the ruling on a day that they knew an objecting councilor would not be present. County clerks were instructed to waive the three-day waiting period and the 60 license applications a day soared to 400.  Given the overwhelming numbers and the fact that many ministers would not officiate, there was a call for ministers to perform the ceremonies. On March 4, 2004, I called up a woman I was due to have a blind date with the next day and informed her I would be cancelling it because I had to help. She was delighted with my excuse (and we later started dating).  On March 5th, I found myself standing on the sidewalk outside the Multnomah County Courthouse, wearing a nice suit with a clerical collar. I had bought the collar years ago for a Halloween party and was happy to discover that it was a real collar from a ministerial supply company.    The Multnomah County Courthouse Source    One couple I married had been together for eight years and another for seventeen. Their joy at finally being married was wonderful. They cried, they laughed, they hugged and kissed. It was a great thing. I can’t understand how anyone could tell them “no.”  Only one couple gave me pause, a couple of boys who looked eighteen. I wasn’t bothered by the fact that they were gay, of course. It was just that they were so young and I was worried they weren’t mature enough to be married, much less face the discrimination that they were going to face, as evidenced by people standing around with signs announcing that we were going to burn in hell. However, their mothers were with them, beaming with pride. Of course I officiated.  As for myself, I should have brought a lot more water with me. My throat was getting scratchy. I also should have brought some Advil because I found that standing on the sidewalk officiating at wedding after wedding was painful. I was cold and was getting a bit rummy at the end. I stared at some of the forms and once had trouble figuring out what my name was over the shouting of protestors.  There were loud cheers for the first wedding, trickling down to a handful near the end of the day. I married seven couples that day and it was awkard at times. I would ask their names and carefully write them down on their copy of the vows I had hastily written the night before. I had already learned my lesson about forgetting names. It was also mortifying to ask, “For purposes of filling out the paperwork, who will be the groom?” Unsurprisingly, the paperwork hadn’t been updated for same-sex couples.  I also needed two witnesses for every wedding and some couples were there alone, no family or friends to support them. I had no trouble finding volunteers in the crowd.  When the day ended, I was approached by another minister who was also dead tired. He explained that he had secured a free banquet room for the following day and had arranged to have couples sent there for their ceremonies. He saw my ceremonies and asked if I could come help. I was delighted to.  Somewhere out there is news footage from a local television station filming me doing weddings on the sidewalks. I had several calls from excited friends who congratulated me. I wish I could find that footage.   The next day I showed up at the restaurant and throughout the course of the day, the other minister and I would trade off batches of weddings. I performed ten of them that day. Most were lovely and the couples were often crying at the end. One was horrendous when one of the men was very belligerent, demanding to know why I was wearing a clerical collar. I pointed out that I was doing him a favor and his partner quickly calmed him down, but the man glared at me throughout the ceremony. I was tempted to stop and hand them off to the other minister, but his partner looked so embarrassed that I continued.  While waiting for my next batch of weddings, I found myself talking to a delightful lady who was terribly excited that she could finally get married to the love of her life. We had a great conversation and when it was her turn to get married, she asked the other minister if I could officiate instead. It was a lovely ceremony and after I finished, she insisted I come to their reception. I was happy to accept, and she gave me the address of a place nearby that I had never heard of. As I looked at the address, I noticed she included her phone number, “in case I got lost.” I pocketed the address as she was explaining to me that she was actually bisexual and she and her partner had an “open” relationship. And then she just looked at me and smiled.  Oh, shit.  I might be open-minded, but I was not going there. However, I had already agreed, in front of the wedding party, to attend the reception. I made my excuses, found the address, and walked in the front door of a smoke-filled dive bar. Several patrons glanced over, trying to figure out why a minister would be standing there. I had forgotten to remove my collar.  Screw it. I stumbled up to the bar, lit a cigarette and ordered a whisky. This was Portland and the bartender didn’t bat an eyelash. The wedding party arrived, were having a great time, and as soon as I felt I could escape, I made my excuses and left. I never did call that phone number.  The BBC  So those were my twenty-one weddings, only one of which survived. A little over one month later, the Oregon Supreme Court annulled the almost 3,000 same-sex weddings  on the grounds that only the state legislature has the power to regulate marriage. Once again, right and wrong only had a passing flirtation with legal and illegal. My seventeen same-sex marriages were over. I was crushed, but that’s nothing compared to how many lives were devastated by that ruling.  And that is how, in 2010, I found myself working at the BBC in London, trying to deal with a rather awkward phone call.  I was moving in with Leïla, my then-fiancée. She had offered to pack my flat in Pimlico and then we’d move all of the boxes to her flat in Finsbury Park. Leïla is methodical. Organized. Careful. She went through all of my belongings, very carefully, and sorted them into appropriate boxes. She does this every time we move to ensure that when we unpack, nothing is misplaced.  As she was packing my belongings, she stumbled across some paperwork. She’s French and at the time, her English was good, but not fluent. My new fiancée had called me at work, quite unhappy, to know why she found seventeen wedding certificates with my name on them. Was I a bigamist?  We still laugh about that to this day.  Update : I’ve since been reminded of another wedding I officiated at. Oops. That makes 22 of them. That marriage has also ended, so excluding my current marriage, I am proud of say I have a 100% track record of having presiding over failed marriages.  ","title":"My 21 Weddings","url":"\/blog\/my-21-weddings.html"},{"body":"  Obviously, the title is hyperbolic. Not all black people hate white people’s dreadlocks, cornrows, or other stereotypically black hairstyles, but it’s trivial to hit your favorite search engine and read why black people get mad when white people have dreadlocks  . This is a non-issue for many black people, but for others, they’re extremely passionate about it. Why?    Is this cultural appropriation? Source    A couple of years ago, I wrote elsewhere about this topic, but from a slightly different point of view. When white people strongly reject black culture, that’s racism. But when we admire parts of black culture and try to incorporate them, it’s cultural appropriation! Damned if we do, damned if we don’t. Worse, pointing out that some of the earliest depictions of dreadlocks are amost 6,000 years old, from the Minoan civilization in modern-day Greece  can inflame the argument. Making that point often seems tone deaf to the issues the black person is concerned about.  Yet, that’s more or less the argument I made (this is from memory; details are probably off). While my (mostly white) friends generally seemed supportive of this point of view, a black friend of mine said something to the effect of “that’s what a white person would say.” They never responded when I asked what they meant. I really wanted to understand, but it didn’t happen that day.  Today, the Black Lives Matter movement has inserted itself strongly in US politics . Discussions of institutional racism, defunding the police, and similar matters are pervading the American consciousness again, in a way that perhaps wasn’t there since the Civil Rights Movement  . Unsurprisingly, these topics are still terribly divisive. For some, these discussions are cause for them to double-down on their insistence that these issues are overblown. For others—myself included—these discussions have led down a rabbit hole of learning.    My wife’s hair, like my wife, is beautiful.   Unfortunately, the wording of some of these topics makes it even more confusing. For example, in one video when Ben Shapiro is asked about “institional racism”, he immediately demanded to know which [institution] is the racist one?  . It’s a brilliant debating tactic, even if it is fundamentally dishonest. Institutional racism isn’t “that organization down the street hates black people.” Hell, it’s not even racism in the sense that most people think of it. Institutional racism is the result of fundamentally unequal outcomes experienced by different ethnic groups due to structural issues in society’s laws and culture. But that doesn’t roll off the tongue as easily as “institutional racism,” so people like Ben Shapiro nitpick the latter term and hope you won’t notice the bigger picture.    And don’t even get me started on the mess of the slogan “Defund the Police.” It was never about defunding the police and of course it gives many people the idea that we’re arguing for anarchy. For those who are disinclined to listen in the first place, messages like “defund the police” drive us apart instead of together. Even former president Barack Obama has warned against the phrase  , to little effect.  Yet, in the face of the protests and the issues surrounding it, hair seems like a minor issue, right? Well, not it’s not minor for many black people. Chris Rock produced an entire documentary about the black experience with hair.     Simply put, black hair is often different. It’s often tightly curled and considerably more difficult to brush and style than white hair. There’s a culture around black hair and it’s strongly associated with black cultural identity. In 2014, when the US Army banned female soldiers from wearing cornrows, braids, twists and dreadlocks  , they didn’t mention race, but it was clear it was about race. With the limited options that black people have to style their hair, the US Army decided to take those options away. Or there was the case of Vanessa VanDyke. Her school threatened to expel her because of her hair  . It was natural. There were no cornrows, braids, twists, or dreadlocks. It was simply black hair.  Many black people can tell you stories about being told their hair is dirty, or unprofessional, or offensive. People use black hair as an excuse for discrimination. Black hair is “ugly.” And yet when white people wear traditionally back hairstyles, they’re “hip” or “cool.” Usually the grief they get seems to be from black people, not white. Why should white people get away with something black people aren’t allowed to get away with?  I still can’t say that I know what the right answer is here, but the wrong answer is to fail to understand why hair can be such a sensitive topic to black people. I had no idea.  But I think it’s even better summed up in this comment on Reddit  .   There is a long history in the US of black folks being othered for their hairstyles. Braids, locs, natural afros, super tight curls, all these things have set black Americans apart from white Americans. Even though protective hairstyles like braids and locs are necessary to keep many black hair textures healthy, the look was different than white people and therefore “unprofessional” and unattractive. Only in the last few decades have we begun to undo this conditioning. If you want a way better and way more compelling explanation than mine, Chris Rock made a documentary called “Good Hair” to explain black American culture around hair and it’s fascinating.  Anyway, if you’re from a group who has been historically criticized for the way your hair grows out of your head and then see some white people being able to do the same thing with relatively no scrutiny, you’d be pissed too. Cultural appropriation is not simply borrowing things from other cultures, it’s borrowing them without acknowledging what it means to that culture. In the US, black othering and the existence of “good hair (white hair)” is much more relevant to our culture than Viking matting.   ","title":"Why Black People Hate Your Dreads","url":"\/blog\/why-black-people-hate-your-dreads.html"},{"body":"    George Floyd did not deserve to die. Source    Strange fruit on red lines cross blue lines and die.  In the last moments of George Floyd’s life, he was lying on the ground, a terribly symbolic knee on his neck, begging for his life and crying for his mother. It was heartbreaking.  And it was broadcast across the world.  For those who are not Americans, you might wonder how the death of this man has led to riots, has led to the President of the United States threatening to shoot those responsible for “looting” (A threat which, if carried out, violates the oath of office Trump swore to uphold  ).  But it’s shouldn’t be hard to understand. Colin Kaepernick lost his career and was vilified in the press for respectfully “taking a knee” during the National Anthem.   Yet another uppity black person who didn’t know his place. You can’t protest peacefully. You can’t vote your way to equality if enough people vote against you. You can’t protest racism without being accused of being racist  . What avenues are for protest are left?  Strange fruit on red lines cross blue lines and die.  Strange Fruit  , started as a poem and later became that magnificent song by Billie Holiday singing about black people being lynched in the south and hanging from trees (like fruit). Listen to Jill Scott singing Strange Fruit.     Redlining  , of course, is the historical US practice of keeping black people out of white neighborhoods.  The blue line, of course, represents the police.  In a recent study of US police use of force  , we find that black people are two and a half times more likely to be killed by police than white people. The reasons are debated  , but it’s a moot point. As The Guardian found in their comprehensive research of US police killings, US police routinely kill more people in days than other countries do in years  . From that article:   In the first 24 days of 2015, police in the US fatally shot more people than police did in England and Wales, combined, over the past 24 years.   You can argue all you want about why police are killing more black people than white people, but this neatly sidesteps the issue of why US police are killing at all.  Despite that, I’ll sidestep that issue. I’ll even sidestep the issues of racial justice because there should be no need to belabor the point that being black in America is fraught with danger and discrimination. Instead, I want to point out where this could be leading.  The New Civil Rights Movement  The Civil Rights Movement  of the 50s and 60s was people of all skin tones working together to end the widespread discrimination against blacks. The 14th Amendment  guaranteed all US citizens “equal protection of the laws” and the 15th Amendment  gave black people the right to vote.    Thomas Mundy Peterson was the first black person to vote after the passage of the 15th amendment. Source    After the 15th Amendment was passed, Thomas Peterson was the first black person to vote in the United States. The people in his town later raised money and gave him a gold medal celebrating this event. However, Jim Crow laws  , mostly enacted in the Southern US, were passed with the goal of taking away this right.  These laws were largely successful. Redlining ensured that black people were not allowed to live in “white” communities and would be denied the economic opportunities that white people enjoyed. Education systems funded by property taxes meant that white children received better education than black children, which, in turn, helped to ensure that black people in the US would be economically disadvantaged.  But all of that pales in comparison to the thousands of black people who were lynched in the (mostly Southern) US  , almost universally without legal repercussions. According to the NAACP  :   From 1882-1968, 4,743 lynchings occurred in the United States. Of these people that were lynched 3,446 were black. The blacks lynched accounted for 72.7% of the people lynched. These numbers seem large, but it is known that not all of the lynchings were ever recorded. Out of the 4,743 people lynched only 1,297 white people were lynched. That is only 27.3%. Many of the whites lynched were lynched for helping the black or being anti lynching and even for domestic crimes.     The Lynching of Tom Shipp and Abe Smith at Marion, Indiana, August 7, 1930 Source    These and many other issues helped drive the Civil Rights Movement, but today, many Americans don’t realize just how bloody that movement was. Sure, we know about the assassinations of Malcom X and Martin Luther King, Jr. And maybe you’ve a vague recollection of someone named Medgar Evers  , but did you also know about George W. Lee  , Lamar Smith  , Dr. Thomas Brewer  , Herbert Lee  , William Moore  , Louis Allen  , James Chaney  , Andrew Goodman  , Michael Schwerner  , James Reeb  , Viola Liuzzo  , Jonathan Daniels  , Sammy Younge Jr.  , Vernon Dahmer  , Robert W. Spike  , or Wharlest Jackson  ?  No, you probably didn’t know most of those names (I certainly didn’t). The US school system largely omits the history of the Civil Rights Movement  and to this day, it’s hard to get a full accounting of how many people died as a result of that movement, but in reading their stories, one thing becomes clear: many of them knew they might die, but they did what they did anyway. But we usually didn’t learn that in schools; it’s just not taught.  It’s a crime that we’re not taught about all of those people murdered for the crime of thinking that discrimination against black people is wrong.  These issues drove the Civil Rights Movement of the 50s and 60s, but George Floyd’s death, or the future death of another person like him, will drive the new Civil Rights Movement.  Thugs  I recently shared a video on Facebook that went viral  and currently has over 20 million views. I was flooded with friend requests. So I decided to conduct an experiment. With the exception of those accounts who clearly appeared to be bots, I accepted all of those friend requests in an attempt to positively engage with those who had different worldviews from mine. My wife wasn’t happy about this and, I must admit, she was right. I learned quite a bit from that experiment, but the main takeaway was simply that Facebook isn’t the place to have nuanced discussions about political issues.  The latest trainwreck of conversation was by someone reading one of my posts about George Floyd’s death and the subsequent riots and simply replying “Oh well thugs are thugs after all.” Their subsequent comments flowed into anti-LGBT territory and a defence of Trump, just in case anyone misunderstood where their sympathies lay. These attitudes by those who ”friended” me on Facebook are very, very common.  Black people aren’t protesting against the death of George Floyd. They’re just thugs. And what are these thugs protesting against?  Christian Cooper (who happens to be black), a board member of the Audobon Society in New York, was bird watching when he asked a woman to leash her dog in an area where dogs weren’t allowed off leash. Her response (on camera) was to yell “I’m going to tell them there’s an African American man threatening my life!”  Ahmaud Arbery, an unarmed black man out for a jog, was hunted down and killed by three white men in Georgia.  Trayvon Martin, a black teenager, was shot just for walking home.  Botham Shem Jean, also black, was relaxing in his own apartment when he was murdered by an off-duty police officer.  Oscar Grant (coincidentally, black), was held down by police and shot in the back.  In fact, one study of Deaths Due to Use of Lethal Force by Law Enforcement  found:   Victims were ... disproportionately black with a fatality rate 2.8 times higher among blacks than whites ... [Black] victims were more likely to be unarmed than white or Hispanic victims.   Black children have been shot and killed by police for carrying toy guns or cell phones. Eric Garner was choked to death by police  for selling cigarettes. In 2015, an average of two unarmed black people were killed each week by police in the United States  .  A new Civil Rights Movement is starting in the United States.  It’s not about disenfranchisement.  It’s not about redlining.  It’s not about loss of job opportunities.  It’s about being murdered for being black.    Black Lives Matter   There’s a new civil rights movement brewing in the United States of America and it’s long overdue.  ","title":"The Murder of George Floyd and The New Civil Rights Movement","url":"\/blog\/the-murder-of-george-floyd-and-the-new-civil-rights-movement.html"},{"body":"  I’ve been wanting to get back to playing around with AI life simulation. Years ago I was writing a small simulation in Perl. I was going to “evolve” tigers, cows, and grass in a simple food-chain simulation.  The tigers would learn to hunt cows, the cows would learn to eat grass and avoid tigers, and the grass, well, the grass would grow.  I wrote a lot of code. The brains were simple “winner take all” neural networks and the animals could eat, move forward, or turn.  What sort of amazing strategies would these animals evolve? After they evolved, what sort of new animals and constraints could I add to make the simulation more interesting?  I had written plenty of code, lots of tests, and was quite proud of my work. Except it didn’t work.  The cows wandered around aimlessly. The tigers wandered around aimlessly. The grass grew. My animals would not evolve.  I wrote more tests. Maybe they couldn’t see? No, they could see. Maybe eating wasn’t giving them enough energy? No, eating was giving them enough energy.  Maybe their brains didn’t work? No, their brains worked just fine. Mine, however, didn’t. I was stumped.  I kept poking and prodding and theorizing and moaning and finally had an epiphany. I had, once again, written unit tests for everything.  I hadn’t written integration tests. Amongst other things, I teach software testing and over and over again, I warn people how critical integration tests are. But here I was, making the same mistake.  So I wrote integration tests. And that’s when I discovered it.  My animals could think. My animals could see. But I never connected their eyes to their brain. They never thought about what they were seeing.  I had written a voter simulator.  ","title":"My AI Life Simulator","url":"\/blog\/my-ai-life-simulator.html"},{"body":"    Corinna’s First (mis)Steps  Good OO Design  My Mistake  Politics  Class Data and Methods  Inclusion  Semantics  Syntax  State Variables  The Future  Conclusion  Update     Corinna’s First (mis)Steps  Note: Corinna is the name of Ovid’s love interest in the Amores  .   I had been working on the design for Corinna, a proposed object system for the Perl core  , for several months, inspired by the work of Stevan Little  , pulling ideas from many other sources, including far too much reading about effective object-oriented design. Sawyer X  , then the Perl pumpking (a now defunct role, but it was the person overseeing Perl language development), urged me to stop working on the implementation and just focus on the design. Design something great and he’d find someone to implement it.  When I first announced Corinna (then known as “Cor”), it was met with enthusiasm, indifference, and hostility, depending on whom you asked. In fact, I was convinced that what I had proposed was solid enough that I was sure the community would buy into it easily. They did not.  Truth be told, they were right to not get excited. Ignoring the fact that people have wanted to get an object system in the Perl core for years and no one had succeeded, the fact is that Corinna wasn’t really that interesting in terms of an OO system for the Perl community. I had spent so much time researching good OO practices that I hadn’t take into consideration the impact of decades of ingrained behaviors in the Perl community. Here’s what an early version looked like.  class Cache::LRU v0.01 { use Hash::Ordered; has cache =&gt; ( default =&gt; method { Hash::Ordered-&gt;new } ); has max_size =&gt; ( default =&gt; method { 20 } ); method set ( $key, $value ) { if ( self-&gt;cache-&gt;exists($key) ) { self-&gt;cache-&gt;delete($key); } elsif ( self-&gt;cache-&gt;keys &gt; self-&gt;max_size ) { self-&gt;cache-&gt;shift; } self-&gt;cache-&gt;set( $key, $value ); } method get ($key) { if ( $self-&gt;exists($key) ) { my $value = $self-&gt;cache-&gt;get($key); $self-&gt;set( $key, $value ); # put it at the front return $value; } return; } }  I won’t bore you with the details of the bad decisions I made, but you can read the initial Corinna specification here  . Let’s look at one mistake I made in that specification:   Note: “slots” are internal data for the object. They provide no public API. By not defining standard is =&gt; 'ro' , is =&gt; 'rw' , etc., we avoid the trap of making it natural to expose everything. Instead, just a little extra work is needed by the developer to wrap slots with methods, thereby providing an affordance to keep the public interface smaller (which is generally accepted as good OO practice).   So you couldn’t call $self-&gt;cache or $self-&gt;max_size outside of the class.  Good OO Design  So I said encapsulation is “generally accepted as good OO practice.”  Yes, that’s true. And if you’re very familiar with OO design best practices, you might even agree. Java developers learned a long time ago not to make their object attributes public, but Perl, frankly, didn’t really have simple ways of enforcing encapsulation. In fact, in the early days of OO prorgamming in Perl, you’d often see an object constructed like this:  package SomeClass; sub new { my ( $class, $name ) = @_; bless { name =&gt; $name, _items =&gt; [], }, $class; } 1;  With that, even outside the class you could inspect data with $object-&gt;{name} and $object-&gt;{_items} . But why the leading underscore on _items ? Because that was a signal to other developers that _items was meant to be private. It was asking them to not touch that because Perl didn’t offer an easy way to ensure encapsulation.  Something similar for Moo\/se could be written like this:  package SomeClass; use Moose; has &#39;name&#39; =&gt; ( is =&gt; &#39;ro&#39;, isa =&gt; &#39;Str, required =&gt; 1, ); has &#39;_items&#39; =&gt; ( is =&gt; &#39;ro&#39;, isa =&gt; &#39;ArrayRef&#39;, default =&gt; sub {[]}, init_arg =&gt; undef, ); 1;  That almost the same thing, but now you have a reader for $object-&gt;_items and you still don’t get encapsulation. Bad, right?  Well, yeah. That’s bad. All Around the World  has repeatedly gone into clients and fixed broken code which is broken for no other reason than not respecting encapsulation (drop me a line if you’d like to hire us).  There have been attempts to introduce “inside-out” objects  which properly encapsulate their data, but these have never caught on. They are awkward to write and, most importantly, you can’t just call Dumper($object) to see its internal state while debugging. That alone may have killed them.  My Mistake  So if encapsulation is good, and violating encapsulation is bad, why were people upset with my proposal?  Amongst other things, if I wanted to expose the max_size slot directly, I had to write a helper method:  class Cache::LRU v0.01 { use Hash::Ordered; has cache =&gt; ( default =&gt; method { Hash::Ordered-&gt;new } ); has max_size =&gt; ( default =&gt; method { 20 } ); method max_size () { return self-&gt;max_size } ... }  If I wanted people to be able to change the max size value:  method max_size ($new_size) { if (@_ &gt; 1) { # guaranteed to confuse new developers self-&gt;max_size($new_size); } else { return self-&gt;max_size; } }  And there are many, many ways to write the above and get it wrong. Corinna is supposed to make it easier to focus on writing good OO code. Here, it was letting developers fall back to writing bad code that Corinna could easily write. Why should people have to write it? If I offered some convenience here, how should it look? Enough people argued against the design that I realized I made a mistake. Maybe encapsulation is good, but my strictness was over the top. Which leads me to the entire point of this blog entry:  If you’re not effective, it doesn’t matter if you’re right.  Was I right to strive for such encapsulation? I honestly don’t know, but if it meant that no one would use my OO system, it didn’t matter. I had to change something.  Politics  It was pointed out to me that since I had been working in isolation, I hadn’t had the chance to hear, much less incorporate, the views of others. It was time for politics.  Out of all of the online definitions about politics, I really like the opening of Wikipedia  the best:   Politics is the set of activities that are associated with making decisions in groups ...   That’s it. People often hate the word “politics” and say things like “politics has no place in the workplace” (or community, or mailing list, or whatever). But politics is nothing more than helping groups form a consensus. When you have a group as large as the Perl community, if you’re not interested in forming a consensus for a large project, you’ve already put yourself behind the eight ball and that’s what I had done.    Source    So an IRC channel was created (irc.perl.org #cor) and a github project  was started and anyone with an interest in the project was welcome to share their thoughts.  To be honest, this was challenging for me. First, I don’t respond well to profanity or strong language and there were certainly some emotions flying in the early days. Second, nobody likes being told their baby is ugly, and people were telling me that. For example, when I was trying to figure out how to handle class data, here’s part of the conversation (if you leave a comment, I will delete it if it’s identifying this person):   developer: Ovid: this is not an attack on you or at all, I’m just about to rant about class data, ok  developer: class data is bullSHIT  developer: there is no such thing  developer: it’s a stupid-ass excuse in java land for them not having globals  developer: just make it a fucking global  developer: or a fucking constant method  developer: sub foo { ‘value' }  developer: there, it’s a constant method, it’s fine, fuck off   When I see things like that, I tend to tune out. That’s not helpful when I need to understand where people are coming from. And I’ve read that multiple times and I still don’t see the technical argument there (to be fair, this went on for a bit, but I saw nothing explaining why global variables are better than class data. It could just be that I’m a bear of little brain).  I strongly disagree that globals are better than class data because I’ve worked on codebases with plenty of globals and, even if they’re immutable, there’s the lifecycle question of when the come into existence. Plus, class data is clearly associated with a class, so at the very least, when I call my $foo = SomeClass-&gt;data (assuming it’s needed outside the class), the class can maintain responsibility for what that data is. Or if the class data is private to the class (as it often is), a global breaks that entirely. But nonetheless, I have to get buy-in for Corinna and that means listening to people and trying to figure out what the real complaints were and how (and whether) to address them.  This worked well for slot encapsulation because we eventually came up with this:  slot $name :reader :writer;  The $name variable can be read via $object-&gt;name and can be set via $object-&gt;set_name($new_name) . Yes, this tremendously violates encapsulation because Corinna does not yet have a way of enforcing type constraints, but mutability at least isn’t the default. And when we later incorporate type constraints, the external interface of the classes won’t have to change. I got buy-in for the design, at the cost of slightly compromising some of my design goals. I think it’s a good trade off and I think Corinna is better off for this.  But what about class data?  Class Data and Methods  Should we include class data and methods in Corinna? I discovered that for many people, the answer was obvious, but their obvious answers disagreed with other people’s obvious answers. What makes it worse is that this isn’t just a “yes or no” question. We have multiple questions.   Should we include class data and methods?  What should the semantics be?  What should the syntax be?   This isn’t easy. We got past one and two with difficulty, but number three has been a bit of a trainwreck. And I’m the conductor.  Inclusion  I think it was pretty much universally agreed that we had to have class methods. These wouldn’t have $self injected into them, nor would they have any access to instance data. In fact, done properly, we could make violations of that a compile-time failure. That’s a huge win for Perl. They might look like this:  slot $some_data; # instance data common method foo() { # $class is available here # $self and $some_data are not available here }  One use of those is for alternate constructors. In Moo\/se, you have BUILDARGS for fiddling with arguments to new . In Corinna, at least for the MVP, you write an alternate constructor (we don’t have method overloading in Perl).  class Box { slot ( $height, $width, $depth ) :param; slot $volume :reader = $height * $width * $depth; common method new_cube ($length) { return $class-&gt;new( height =&gt; $length, width =&gt; $length, depth =&gt; $length, ); } }  With the above, you can create a cube with Box-&gt;new_cube(3) . No more messing around with BUILDARGS and trying to remember the syntax.  So if we should have class methods, should we have class data? Well, it’s been pointed out that it’s literally impossible to avoid:  class Box { my $num_instances = 0; # class data!! slot ( $height, $width, $depth ) :param; slot $volume :reader = $height * $width * $depth; ADJUST { $num_instances++ } DESTRUCT { $num_instances-- } common method new_cube ($length) { return $class-&gt;new( height =&gt; $length, width =&gt; $length, depth =&gt; $length, ); } common method inventory_count () { $num_instances } }  So at a very practical level, whether or not we want class data, we have it. In fact, even if we omitted class methods, we’d still have class data. So let’s work with what we have.  Semantics  We have a lot of the semantics described here  , but it could use some more work. However, the general idea of behavior isn’t controversial enough that I want to spend too much time here.  Syntax  Now we have a problem. In Moo\/se, we BUILD , DEMOLISH , and has , which have been renamed to ADJUST , DESTRUCT , and slot in Corinna in part because they’re different beasts. We don’t want to have BUILD in Corinna and have Moo\/se developers think it’s the same thing. If we get an analog to BUILDARGS , it will probably be called CONSTRUCT for the same reason.  So one of our design goals is that different things should look different.  Another design goal is that we do not wish to overload the meaning of things. Thus, we agreed that reusing the class keyword ( class method foo() {...} or class slot $foo ) was probably a bad idea (it turns out to be spectacularly bad if we get to inner classes, but let’s not go there yet).  By the same reasoning that “different things should look different,” similar things should look similar. In Java, class data and methods are declared with the static keyword.  public class MyClass { private String name; \/\/ class data public static int numberOfItems; public MyClass(String name) { this.name = name; } \/\/ class method public static void setSomeClassData(int value) { MyClass.numberOfItems = value; } }  A developer can easily understand how the two are analogous. But do we need this for Corinna? Here’s the “accidental” class data we could not avoid.  class Box { my $num_instances = 0; # class data!! slot ( $height, $width, $depth ) :param; slot $volume :reader = $height * $width * $depth; ADJUST { $num_instances++ } DESTRUCT { $num_instances-- } common method new_cube ($length) { return $class-&gt;new( height =&gt; $length, width =&gt; $length, depth =&gt; $length, ); } common method inventory_count () { $num_instances } }  But we could get rid of inventory_count method by supplying a reader (and even renaming it).  class Box { my $num_instances :reader(inventory_count) = 0; # class data!! slot ( $height, $width, $depth ) :param; slot $volume :reader = $height * $width * $depth; ADJUST { $num_instances++ } DESTRUCT { $num_instances-- } common method new_cube ($length) { return $class-&gt;new( height =&gt; $length, width =&gt; $length, depth =&gt; $length, ); } }  So right off the bat, for new developers, we need to teach them when they can and cannot use slot attributes with my variables.  Also, Perl has the conflation of package and class , along with sub and method . Do want to add my for class data and for lexical variables? And as Damian Conway has pointed out  , static analysis tools are already hard enough to write for Perl, given the overloaded meaning of many keywords.  And if we do accept the notion that similar things should look similar, why would class methods and class data have different declarators? We can’t just say my method foo() {...} because that clearly implies it’s a private method.  Or we can adopt the approach other OO languages such as Java, C++, C#, and Swift have done and use a single keyword to explain the same concept: these things are bound to the class and not an instance of the class. For the aforementioned languages, that keyword was static , but it was strongly shot down as “not good for us” due to possible confusion with the state keyword and the fact that different languages sometimes use static to mean different things. Different things should look different.  shared seems good, but that implies threads to many people  , so that was also shot down.  I’m not sure who came up with the word common (it may have been me back in February of 2021, according to IRC logs) and so far it seems like the least-bad alternative. (Another suggestion I proposed at that time was mutual )  However, there are those who are strongly in favor of my , including adding attributes to it—if it’s in Corinna and not inside a method—and strongly object to common on the grounds that all methods defined in a class are common to every instance of that class. They have a point about common being a poor choice, but I don’t have a good one and I suspect that, over time, it won’t even be noticed (I may live to regret typing that).  So while I’m trying to figure all of this out, Damian Conway posted an extensive endorsement of Corinna  . To illustrate one of his points, he shared a class written in Dios, which an OO system for Perl which he introduced in his “Three Little Words” presentation.     He wrote the following class.  use Dios; class Account { state $next_ID = &#39;AAA0001&#39;; has $.name is rw is required; has $.balance = 0; has $.ID = $next_ID++; method deposit ($amount) { $balance += $amount; } method report ($fh = *STDOUT) { $fh-&gt;say( &quot;$ID: $balance&quot; ); } }  That’s almost how you would write it in Corinna, but that’s not what I really noticed.  I kept staring at that state variable he used to declare class data.  Everyone arguing for reusing an existing declarator to declare class data in Corinna was arguing for my . Here’s Damian, using state .  I couldn’t get that out of my mind. And then I started thinking about inner classes, but let’s not go there yet. Let’s talk about state and why this is important.  State Variables  perldoc -f state  says:   state declares a lexically scoped variable, just like my. However, those variables will never be reinitialized, contrary to lexical variables that are reinitialized each time their enclosing block is entered. See “Persistent Private Variables” in perlsub  for details.   What does that mean? Well, first, let’s run the following code.  sub printit { state $this = 1; my $that = 1; $this++; $that++; say &quot;state $this and my $that&quot;; } printit() for 1 .. 3;  That prints out:  state 2 and my 2 state 3 and my 2 state 4 and my 2  As you can see, state variables are like static variables in C. They are declared once and retain their value between calls. They kinda look like static members in Java.  Let’s look at state some more, this time returning an anonymous subroutine with the variables.  sub printit ($name) { return sub { state $this = 1; my $that = 1; $this++; $that++; say &quot;$name: state $this and my $that&quot;; } } my $first = printit(&#39;first&#39;); my $second = printit(&#39;second&#39;); $first-&gt;() for 1 .. 3; $second-&gt;() for 1 .. 3;  And that prints out:  first: state 2 and my 2 first: state 3 and my 2 first: state 4 and my 2 second: state 2 and my 2 second: state 3 and my 2 second: state 4 and my 2  Hmm, perldoc -f state says that state variables are only initialized once, but in the case of returning an anonymous sub, we’ve created a new lexical scope and we have a different state variable.  Just for completeness, let’s define those variables inside the outer sub, but outside the inner sub.  sub printit ($name) { state $this = 1; my $that = 1; return sub { $this++; $that++; say &quot;$name: state $this and my $that&quot;; } } my $first = printit(&#39;first&#39;); my $second = printit(&#39;second&#39;); $first-&gt;() for 1 .. 3; $second-&gt;() for 1 .. 3;  And that prints out:  first: state 2 and my 2 first: state 3 and my 3 first: state 4 and my 4 second: state 5 and my 2 second: state 6 and my 3 second: state 7 and my 4  So, depending on how we declare those variables and what the enclosing scope should be, we get different results. This is more or less as expected, though creating a new lexical scope and having the state variables re-initialized might surprise some because I don’t think it’s clearly documented.  But what does that mean for class data?  Part of my job is to ensure that Corinna doesn’t break existing Perl. However, I need to ensure that Corinna doesn’t hobble future Perl, either. That’s much harder because we can’t predict the future.  The Future  There are two things we would love to see in the future for Perl. One is inner classes and the second is anonymous classes. “Anonymous classes” already feels “Perlish” because we have anonymous subroutines and most Perl developers are familiar with the concept of closures. But let’s discuss inner classes first since many people are not familiar with them. Let’s look at some examples from the Java documentation  .  class OuterClass { ... class InnerClass { ... } static class StaticNestedClass { ... } }  The InnerClass has access to all static (class) and instance variables in OuterClass , while the StaticNestedClass class only has access to the static variables.  What this means is that you can put together a collection of classes and encapsulate the “helper” classes. When people talk about allowing classes to “trust” one another with their data but not share it with the outside world, this is a way to do that while still maintaining encapsulation.  For Corinna, it might look like this:  class OuterClass { ... class InnerClass { ... } common class StaticNestedClass { ... } }  So we’d immediately have something we can reason about, with well-defined, battle-tested semantics from the Java language (if we’re allowed to steal from other languages, Java should be on that list. No language bigotry, please).  (As an aside, this is why we can’t reuse the class keyword for class data and methods. How would we describe a static inner class? class class StaticNestedClass {...} ?)  Next, let’s consider an anonymous class. Here’s one way to think about it.  my $thing = class { slot $foo; slot $name :param; ... }; my $o1 = $thing-&gt;new(name =&gt; $name1); my $o2 = $thing-&gt;new(name =&gt; $name2);  We could go the Java route and allow declaration and instantiation at the same time  , but I don’t think that gains us anything:  my $object = (class { slot $foo; slot $name :param; ... })-&gt;new(name =&gt; $name1);  But consider this:  class OuterClass { ... private class InnerClass { ... method some_method (%args) { return class { state $class_data = 17; # or my $class_data = 17 slot $name :param; }; } } }  So we have a private inner class which returns anonymous metaclass instances with state or my variables being used for class data. Are they shared across all metaclass instances or not? I would think “no”, but someone else might make a reasonable argument otherwise. And should it be state or my ? Do either really connote “global to this class regardless of how the class is generated”?  And what we’re talking about is something speculative, years in the future, where the existing semantics of my or state might be perfectly appropriate. Or after we get there, we might discover that they’re not appropriate and we’ve backed ourselves into a corner because we decided to reuse a handy thing in the language.  Conclusion  There are no good answers here, but I had to make a call. And I decided to err on the side of established conventions in popular languages, and potential safety for the future of Perl. I also didn’t want to potentially overload the meaning of existing syntax.  There are already people who have let me know that they’re very upset with this decision. There are others who are fine with this decision; they just want to get Corinna in core. In this case, I don’t think there’s a “safe” political solution. So I decided to play it safe technically.  People might come back to me later and make a strong argument that I screwed up. That’s fine. I welcome those arguments and I might change my mind, but the arguments have raged since February with no sign of consensus. I had to make a call. I might not have made the right call, but it needed to be done.  Update  I’ve been getting some feedback on this. I’ve now been going back through tickets, emails, and IRC logs and I see that there some agreement on class method foo () {...} , but a lot of disagreement over class $foo . There was some discussion of class slot $foo being OK. There has been so much discussion of this issue for months on end, on IRC, email, and github, that I’ve not remembered all of the fine-grained detail well.  The use of a class as a both a class declarator and a non-access modifier overloads the meaning of class and violates the “different things should look different” principle (to be fair, it’s not the only place we’ve violated this). And this still doesn’t address the awkward case of declaring a static inner class: class class InnerClass {...} .  Looks like this debate is not done.  ","title":"Politics in Programming","url":"\/blog\/politics-in-programming.html"},{"body":"  It goes without saying that COVID-19, better known as the coronavirus, is dominating the news right now. Plenty of people are worried and there’s plenty of “fake news” out there. However, an article entitled The Coronavirus Is Much Worse Than You Think  , by Samuel Paul Veissière Ph.D. and published in Psychology Today, not only has a terrible clickbait title, but commits one of the worst sins a news organization can commit: lying without lying.    Source    While I’m sure that Veissière didn’t mean to do this directly, I’m nonetheless horrified by this article because it’s making the rounds and I want a handy link to share to debunk the nonsense right now.  The article’s premise is actually sound: you’re very unlikely to contract the coronavirus and, even if you do, you’re unlikely to die from it, so there’s no need to panic.  So far, so good. Except the author makes a terrible, terrible mistake by initially omitting the word “now”.  So let’s look at his initial question:   Why then are so many countries implementing quarantine measures, shutting down their borders, schools, and soccer games for something that is less likely to happen to anyone than drowning in a single year, or even being hit by lightning in one’s lifetime? Why is the stock-market crashing, and why are school and workplace mass emails, news headlines, social media feeds, and face-to-face conversations dominated by stories about what is essentially a new strand of mild to moderate flu?   The error in that should be blatantly obvious, but it would be hilarious (were the situation not so grave) given that the author followed this up with:   To understand this strange dynamic, consider people’s blatant inability to make statistically correct inferences about actual risk ...   OK, let’s talk about statistics. In particular, we’re going to talk about “dependent” and “independent” events.  What are the chances that I contract coronavirus? Right now, they’re pretty small. But what if my wife catches coronavirus? My chances increase dramatically. Thus, the chances that I’ll contract coronavirus are highly dependent on whether or not those around me catch coronavirus. Thus, these are what we call “dependent events” and the chances of one materially effect the chances of the other.  This is what Samuel Paul Veissière, Ph.D., doesn’t seem to understand about statistics. To answer his original question about the quarantine when the risk is so low: the risk can remain low if and only if the quarantine works.  Asshole. (Dr. Veissière is probably actually a wonderful person, but this sort of irresponsible writing really makes me mad).  So the risk is dependent on the quarantine, but let’s talk about independent events because they make the situation worse. My wife and I have a young daughter. Let’s say that one of us catches the disease. The chances are high that all of us will then catch the disease. But the chances of any of us dying from the disease are independent of one another. But what does that mean?  Currently, it appears that the chances of our daughter dying are pretty close to zero. The chances of my wife dying appear to be around 2%, and the chances of my dying are around 4%.   . Well, it turns out that if you do the math , that works out to a 5.9% chance of one of us dying. Instead of a 1 in 50 chance of losing a parent, our daughter has a 1 in 17 chance of losing a parent.  So yeah, coronavirus is a big deal and not something for otherwise respectable publications like Psychology Today to brush off.  But what happens globally? According to Prof. Marc Lipsitch, Professor. of Epidemiology, Harvard School of Public Health, up to 40% to 70% of the world population could get infected.  . He also has very good historical data to back him up. It doesn’t mean this will happen, but if we don’t take preventive measures, it could happen. So what does that mean?  If we’re generous and assume the lower number of 40% of the population is infected, and we assume a 2% (overall) mortality rate, that’s around sixty million dead, or about 170K per day. (To be honest, there’s some hand-waving in this due to unknown rates of infection, re-infection, and so on).  It’s estimated that 150K people die every day (globally), so we’ve more than doubled the number of people dead. But these aren’t people being hit by cars, bullets (go USA!), or dropping dead from a heart attack or stroke. These are people whose deaths are painful and take some time. We’re already overwhelming some of the public health systems. We’re already shutting down businesses all over the world. And that’s long before pandemic status. If we don’t contain the virus, you can expect fewer food deliveries, fewer medical deliveries, more companies failing, and so on. And if you’re one of the unlucky people to contract the virus, you know that probability of dying is significantly higher than it’s ever been.  Oh, and about that 2% chance? While the Dr. Veissière blithely reassures us that we’re all being silly, let’s talk about comorbidities. These are other medical conditions you might have at the same time. In particular, cardiovascular disease, diabetes, high blood pressure, chronic respiratory infections, and other issues appear to dramatically increase your chance of dying from coronavirus.  As a final thought, just in case you think you’re safe, Dr. Veissière does have this to say:   The bad news for you is that, if you live in a densely populated area, you are very likely to contract the coronavirus — if not this year, next year, or the year after as it undergoes its seasonal global migration pattern with its zoonotic cousins.   So there you go. Buried at the end of his article, after he seems to discourage us from quarantines, after he presents a bunch of evolutionary history which isn’t relevant to most people’s concerns, he finally admits that most of us are going to get the disease sooner or later.  COVID-19 certainly isn’t the end of humanity, it’s a very serious problem that absolutely doesn’t deserve this type of sloppy reporting.  Update : it turns out that scientists are also calling BS on this drivel.  Seems that not only was the statistical premise wrong, but the biology was also dead wrong.  ","title":"Coronavirus Fake News","url":"\/blog\/coronavirus-fake-news.html"},{"body":"  Space Exploration Technologies Corp., commonly known as SpaceX, the darling of the launch industry, is one of the most valuable private companies in the US. With almost quarter of all launches worldwide being handled by SpaceX, and with Starship set to revolutionize the launch industry , SpaceX appears to be an unstoppable industry force, but is it?  Eric Berger, the senior space editor at Ars Technica  , posted a bombshell of a tweet.   After the briefing, Administrator Bill Nelson told me that he spoke with SpaceX&#39;s Gwynne Shotwell last weekend. He asked if Twitter\/Musk was going to be a distraction to SpaceX and its commitments to NASA. &quot;She said to me in no uncertain terms, &#39;I assure you it is not.&#39;&quot; &mdash; Eric Berger (@SciGuySpace) December 11, 2022   To put this into context, back in 2019, the Department of Defense (DoD) was investigating whether or not it properly certified SpaceX to provide national security launches  . We do not know why this happened.  Several weeks later, Elon Musk’s security clearance was being reviewed  because he was smoking marijuana on Joe Rogan’s podcast. Even though that was legal under California law, it’s still a Class A misdemeanor under federal law. In fact, three or more possession convictions  can raise this to a felony and this makes it much harder to get a security clearance. If he loses it, he may lose access to the classified material necessary to plan launches for the DoD. That doesn’t mean SpaceX is out, but it would make it harder for him to have an active role.  Then, in October of this year, reports surfaced about Biden administration concerns over the national security implications of Musk’s behavior  . He’s made pro-Russian statements, his reliance on foreign investors to buy Twitter (including Saudi Arabia, Qatar, and a cryptocurrency exchange run by a Chinese-Canadian), and his hints of cutting off Starlink service in Ukraine have all combined to send very confusing signals as to his intentions. The White House later denied those reports of having concerns  .  Shortly after this, SpaceX had a major management reshuffle  , with COO Gwynne Shotwell and vice president Mark Juncosa being brought in to focus on Starship production. Elon Musk is famous for personally handling things like this, but seems to be bogged down by his Twitter antics.  The DoD is in a bind. United Launch Alliance (ULA) are replacing the Atlas 5 rocket with the Vulcan, but Blue Origin has struggled to produce the BE-4 engines needed . Just over a month ago, Blue Origin  delivered the first BE-4 engines, but they’re years late and there’s no evidence that Blue Origin can ramp up production to match SpaceX. Now that Soyuz rockets are no longer an option, the DoD faces the nightmare of SpaceX or nothing. There are many other launch companies, including Firefly Aerospace and Rocket Lab, but they can’t match the cadence, reliability, or launch capabilities of SpaceX. Other rocket companies are either non-US based, or are little more than PowerPoint presentations.  Now the situation is getting worse. The industry needs more launches than launch providers can supply.  Even if Musk ultimately is determined to be a security risk, there’s no viable alternative to SpaceX other than ULA. If Blue Origin can’t deliver enough of the BE-4 engines, it’s SpaceX or nothing. This is not a good position to be in.  This is why Berger’s tweet about NASA is so important. NASA is asking, point-blank, if they can trust Musk. The DoD has had concerns. The Biden administration has (allegedly, but probably true) concerns, and now NASA is asking. So much of the industry is relying on SpaceX and Musk is busy tweeting things like my pronouns are Prosecute\/Fauci  . Not only is this buying into far-right conspiracy theories—something which is sure to put people on edge—but it’s mocking the LGBT community. The latter point probably isn’t an issue from a security point of view, but the QAnon-style tweets, his comments in support of Russia, his support for white supremacists and neo-Nazis on Twitter  are sounding alarm bells.  After months of chaos, Musk seems be begrudgingly admitting that his tenure as head of Twitter may not have been a good thing. He’s let Twitter users vote on whether he runs Twitter and they clearly voted him out.   Should I step down as head of Twitter? I will abide by the results of this poll. &mdash; Elon Musk (@elonmusk) December 18, 2022    Perhaps Musk getting booed off stage  is the wakeup call he needed. Perhaps he couldn’t handle the constant hate thrown at him. Or perhaps he’s realized that running a social media company is harder than he thought.  Whatever the reason, if Musk does step down, it might not happen for months and this won’t change much. He will still own Twitter and whoever takes over his role will be doing what Musk tells him to do. But maybe this will end the daily drama that has accompanied Musk’s takeover of Twitter. This might also allow Musk to give much needed attention to SpaceX.  But I can’t help but shake the feeling that SpaceX might not want that attention.  ","title":"Is SpaceX Stumbling?","url":"\/blog\/is-spacex-stumbling.html"},{"body":"","title":"Why You Will Say \"No\" to Living Abroad","url":"\/blog\/why-you-will-stay-no-to-living-abroad.html"},{"body":"  I always forget many things about basic statistics, but I find I’m more likely to remember them if I write about them. Hence, this blog post. It’s written to help me remember something I keep forgetting, but you can read it too.  Like many of you, I have a near useless weather application on my phone. It tells me one of two things.  First, what is the weather now ? If I’m deep in the bowels of a client’s building in Stockholm or Glasgow, yes, I might want to know the weather now because I don’t know the local climate and glancing at my phone and seeing there’s a blizzard raging is useful.  But my consulting work is usually from home, so I can glance out the window.  The second thing my weather application tells me is the likelihood of rain at 3AM tomorrow morning. Unless I’m planning on being out and about at that time, I don’t care. But my phone assumes I do and I might see something like this:    ...  3AM  4AM  5AM  6AM  7AM ...    ...  13%  13%  18%  22%  8% ...    But what I want to know is whether or not I should send my daughter to school with an umbrella. Thus, I want to know how likely it is that that it will at at all tomorrow. So from the snippet above, is it likely that it’s going to rain between 3 to 7AM? I have no idea.  So how do I calculate that? Well, I am assuming that those percentages are for what are called “independent” events. What’s an independent event? Well, let’s say I want to know the odds of my dying tomorrow. You might assume that I want there to be no chance that I’ll die tomorrow, but that’s not true! I very much want there to be a chance of dying tomorrow because the only way there’s no chance that I’ll die tomorrow is if I’m already dead!  Thus, the odds of my dying tomorrow are dependent on whether or not I’ve survived until tomorrow.  But statisticians are boring and usually don’t use such colorful examples. Instead, they’ll ask a question like “what are the odds of flipping a fair coin twice and getting heads both times?” You simply mulitply the chance of getting heads by the number of times you flip the coin. You can quickly determine that the odds of getting heads twice in a row is 25%. These are also independent events, but they’re not rain.  If we assume that the percentage chances listed for each of those hours don’t take into account whether or not it’s already rained, we have independent event. So, what if, for each of the 24 hours in a day, the odds of rain are 95%? We can safely assume that it will probaby rain at least once tomorrow.  But what if the odds are only 10% for each of the 24 hours? That’s harder to know.  As it turns out, statisticians know very well how to calculate this and they use something called a binomial distribution  , but it’s not useful here because ...  If $p$ is the chance of success of an event, then the chance that we have exactly $k$ successes over $n$ independent trials is:  $$P(X = k) = \\frac{n!}{k!(n-k)!} \\cdot p^k \\cdot (1-p)^{n-k}$$  where $X$ is the number of times the event happened.  But that doesn’t actually help because we don’t want to know the odds of it raining once and only once. We want to know if it happens at least once. And that, it turns out, is easy. The odds of an event happening at least once is 100% minus the odds that it never happened.  The odds of something never happening are the chance of it not happening each hour, multiplied by the other chances of it not happening.  If there’s a 10% chance of rain at 2PM, there’s a 90% chance of no rain. That’s pretty clear, right? But if there’s a 10% chance of rain at 2PM and a 15% chance of rain at 3PM, what are the odds of no rain between 2 and 3PM?  $$(1-.1)\\cdot(1-.15)$$  That works out to roughly 77% chance of it not raining. That means there’s a 23% chance it raining between those two hours, even though the highest chance of rain it 15% at 3PM! Thus, even low percentages quickly add up.  Assuming that every hour has a 10% chance of rain, the chance of it never raining is $(1-.1)^{24}$ (90% multiplied by itself 24 times), or roughly .08, or 8%. Thus, the chance of it raining at least once tomorrow would be 92%.  Even though each hour only has a 10% chance of rain, I still need to send my daughter to school with an umbrella.  ","title":"Will It Rain Tomorrow?","url":"\/blog\/will-it-rain-tomorrow.html"},{"body":"  People do the TL;DR wrong. The summary should be at the top of the wall o' text, not the bottom. How else can you know if you want to keep reading? Maybe that makes it a “Too Long; Don’t Wanna Read.”  TL;DWR: I’ve resigned from The Perl Foundation’s Board of Directors.  This has actually been a long time in coming. In fact, I wrote this a couple of months ago. I started out, many years ago, answering a call for volunteers for The Perl Foundation (TPF). I became a grant manager. Eventually I wound up being the chair of the Grant Committee and at one point, took over our 2005 Google Summer of Code project when every Perl project was failing and I brought it under control with all projects being approved by Google. I was proud of that, though most have never heard of this.  I held on to the role of Grant Committee Chair for a few years before moving to the Steering Committee. Eventually, I wound up on the Board.  Along the way, I moved from the US to the UK, got married, moved from London to Amsterdam, had a daughter, moved to Paris, became a consultant and moved around a bit more. Now I live in the south of France, near the Italian border. Life has been a strange journey and volunteering for TPF has been part of that journey for most of my working life. Walking away from this is not easy.  I was asked by the chairperson of the Board, at one point, if I wanted to be vice-chair. I declined. I was asked (privately) at one point if I wanted to be the chair. I would simply propose a way forward and campaign for the role. I might have succeeded. I might have been the Chairman of the Board of Directors of The Perl Foundation. A prestigious title, even if it doesn’t mean much in the grand scheme of things. But still, that was a bit too much. If you’ll pardon the arrogant comparison, I prefer being Mr. Spock to Captain Kirk. I spent too many years as a child practicing in front of a mirror to raise a single eyebrow to give that up now.  Currently, the Board has members spread over multiple time zones and, in an attempt to find a time that worked for all Board members, the monthly Board meeting was moved to Friday evening, my time. This is the beginning of the weekend for me. It’s family time. It’s important and I don’t want to give it up. I couldn’t attend Board meetings.  I discussed other ways I could still contribute and the situation was understood, but I was falling behind in my understanding of Board issues because so many of them are handled in that meeting. Over time I stopped reading the meeting minutes. And that’s when I realized I was burned out. Or burnt out. Or whatever. I just didn’t care.  I had considered stepping down at that point, but dithered. Walking away from two decades of my life? That’s hard. And I felt like I would also be letting down friends in TPF, many of whom I’ve socialized with in numerous countries. These are people I know better than my friends here in France. This is a painful decision to make.  So let’s talk about the elephant in the room. Or more accurately, the CAT in the room. The controversy over the recent Community Affairs Team (CAT) action, the community’s response to that, the Board’s response to the community response, the community’s subsequent response, and so on … pushed me over the edge.  I don’t want to do this any more.  If you don’t understand the following, consider yourself lucky. I’ve no desire to explain this mess.  To be completely honest, I disagreed strongly with the Board’s final decision, though I voted to abstain, rather than “no.” I voted “abstain” because changes were made and I felt I would seem petty if I kept insisting “my way or the highway.” So I chose “abstain” over “no.” Call me a coward. You won’t be the first.  Did the person in question deserve sanction? Yes. Does the Board have the right to exercise authority over events that it funds\/organizes. Yes. Should the Board have applied the sanction to the individual? No, but only because the rules were not clear at the time of the offending behavior.  So I think what the Board did was wrong. However, the overarching goal of the action was to send a strong message to the Perl community: in those spaces where TPF has authority, TPF will act to ensure a safe environment for all participants. In terms of sending a message, it’s possible that the Board’s action was effective, even if it was wrong. I honestly don’t know the answer, though there’s a very loud contingent of people who will assure you that they do.  I must say that while I disagree with the Board’s actions, I am also disappointed by the response from many people in the Perl community. Some have been repeating untrue rumors, assuming they were true. Others have claimed the CAT team and the Board were acting in bad faith. This makes me sad. Mistakes have been made, but I saw no bad faith. I saw a bunch of good people—people I’ve known for years and consider friends—having to make a decision in a no-win situation. In this case I disagreed with the CAT decision, but I understand why they came to a different conclusion: the Perl Community has ignored toxic behavior for far too long.  To be clear, the CAT decision was more than just what happened at the event or on Twitter. There was a long pattern of inappropriate behavior. Some comments that I personally know about include “I stopped working on open-source project X because of Individual #1” and “I refuse to go on IRC because of Individual #1.” Driving people away from the community is exactly the sort of behavior a standard of conduct is designed to prevent.  Also, it’s true that Individual #1 has done brilliant things for the community. Some of the the most popular, go-to Perl tools are due to Individual #1. I recognize that, but we cannot give someone a pass for that. It wouldn’t be fair.  But I want to make clear that the CAT incident isn’t the reason I’m stepping down. I’m stepping down because it’s time. Fortunately, aside from the CAT incident, the Board is healthier than ever and under Stuart Mackintosh, I think it’s been doing great work. It pains me that the CAT mess overshadows everything else.  I’ve been part of TPF for too long. I’ve read too many profanity-laced emails, some sent privately to me. Some directed at me. I’ve seen horrible toxicity and complaints that we’re “woke” because we don’t want to support that toxicity. Fun fact: calling me “woke” doesn’t insult me; I wear that badge with pride. To me it embodies ideals that I strive for, even if I sometimes (often?) fail to live up to them. I’m trying to be a better person, too.  I’ve also seen incredibly, wonderfully supportive people. The overwhelming majority of encounters I’ve had with people in the community have been positive, if not downright wonderful. And I mean amazingly wonderful. I’ve seen community members struggling and other members reach out with support by job offers, money, or sometimes just kind words given at the moment they’re needed.  I see others who are no longer with the community. They’ve gone on to other things, but still receive love from the community. Still others have been pushed out because they can no longer handle the toxicity, but they’re still loved, too. Others, sadly, have passed on. (It still hurts knowing I’ll never sit in another pub in Belgium, being humbled by Jeff Goff’s understated brilliance, may he rest in peace).  Most people I know in the community are people I’m proud to know. We’re like a family: we disagree and we sometimes fight, but you’re wonderful.  So it’s time for me to stop rewriting this for the umpteenth time, trying to find the right tone. If I’ve learned anything from two decades with TPF, it’s that I can never quite find the right tone. Some people will agree with what I say while others will not. But that’s OK. That’s a community.  I’ve asked the Board if I can join the Advisory Board. It’s a group of ex-Board members who can offer advice (surprise!), but don’t get to vote. I expect they’ll approve, but they might not. Either way, I’m OK with that and I’ll move on and spend time with my wife and daughter, not feeling guilty about missing another Board meeting. I’ll miss many awesome people who volunteer their time supporting a community I love, but it’s time for a change.  Peace.  Update : I should mention that the individual who I said deserved to be sanctioned, but shouldn’t have because the rules were not clear at the time, has accepted full responsibility for their behavior and specifically agreed to the sanctions. They admitted they were in the wrong. I absolutely should have included that bit and I have nothing but respect for their acceptance not only of the decision, but of the reasons for it.  Update 2 : I have deleted comments which named individuals in question. There was a reason I chose not to name anyone.  ","title":"Resigning From The Perl Foundation","url":"\/blog\/resigning-from-the-perl-foundation.html"},{"body":"  What the fuck is that title? It’s not SEO-fodder. It’s not clickbait. It is incomprehensible. Any editor worth their salt would soundly reject this piece on the basis of that title alone. It is not an effective title.  I didn’t write that title for you. I wrote it for me. It’s from some writing of mine from 2007 which, to this day, tickles me pink about a magical time of my life. It’s my favorite title of all time and because it was written for me, it is an effective title.   I am, at heart, a writer. Two published books, one unfinished novel, a finished screenplay, several unfinished screenplays, numerous pieces for various companies, multiple blogs. I write. Even when I spend over a week researching and writing a single article   with only a handful or readers, I’m idiotically proud of it. I write as much for my own edification as for my readers. I don’t lay claim to being a great writer, but here we are, with you, dear reader, still reading.  I also write for pleasure and that is often at odds with writing for teaching...at least if you want your work to be read. If you want the world to find your magnificent crab cake recipe online, you have a boring title like “A Great Crab Cake Recipe,” or “The Best Crab Cake Recipe.” It’s easy for search engines to understand them. A title like “Good Shit In Your Mouth” might make you giggle but probably won’t resonate with many readers.  And as you write for others, blissfully ignoring rules such as “don’t start paragraphs with conjunctions,” you also have to consider if your audience is knowledgeable about your topic. One chef has remarked that when he translates American cookbooks into French, they are invariably shorter because his French audience knows how to cook; it’s their culture. They don’t need step-by-step details on how to sauté something and it would have been a waste of time, not to mention offensive, to explain it.  Another question: will your audience be sympathetic, open, or hostile to your point of view? Far too many writers ignore this. If you write an editorial justifying your belief that a particular president was terrible, the tone of that article can shift dramatically depending on whether that editorial is on the Fox News or The Guardian web sites. It’s the difference between being effective in swaying people or being smugly correct. Do you want to be right or do you want to be effective? You can’t always be both.  Understanding your audience is particularly painful when writing online: you might know your regular readers, but if some of your writing gets widespread, you’ve lost control over your audience and you had damned better well hope that what you wrote can stand on its own.  But there’s another type of writing. A different audience. An audience you know intimately. Yourself. You write for yourself while watching the words erupt from the keyboard in a flurry of paragraphs while you joyfully mix metaphors à la Barbara Kingsolver. Your write for yourself while sneaking a break, luxuriating in the beauty of the words of the late Carlos Ruiz Zafón.   I wrote the above for an unknown audience. I wrote the below for me.  March 2007. I was living in London, though “living” might be a stretch. I was living in a hotel bar with a few dozen of my closest colleagues.  In 2006, I had moved from the US to Nottingham, a small, lovely town a couple of hours north of London. In 2007, the company I was working for announced they were shutting down the Nottingham offices and if we wanted to keep our jobs, we had to move to London. To make a long story very short, the company put all of us up in a hotel for three months—a hotel in Hayes, a suburb of London once described by George Orwell as “one of the most godforsaken places I have ever struck.”  Case in point: shortly after we were herded into this hotel in the middle of nowhere, someone was murdered in the pub across the street from the hotel. The pub was packed, but there were no witnesses. We dubbed it “the murder pub.” We avoided that pub and either we made the long trek into London via the godforsaken Hayes and Harlington train station or we sat in the hotel bar, night after night, drinking. Startled guests would walk into the common area of the hotel to find my colleagues wearing pajamas, watching “footy on the telly” with a lager in one hand and the remote in the other (to prevent people from changing the channel). Living in this hotel is how I came to write the following, entitled “Gustatorial Adumbration and the Sheep of the Universe”.   “Where you from, mate?”  “The U.S.”  “‘Ere on holiday?”  How he missed the first ‘h' and caught the second is one of those mysteries of English speech that, like the game of cricket, is something Americans are Not Meant to Understand.  “No, I live here in London.”  “What’re you doin' in the hotel, then?”  “I sort of live here. It’s a long story.”  And it is a long story. Or a short one. All depends on where you start, really. I’ve spent several weeks at the Heathrow Comfort Inn Hotel, but I’m basically living in a bar with two or three dozen of my closest coworkers. I can’t really count them. Sometimes they’re here. Sometimes they leave for the weekend to go home to Nottingham, Beeston, or wherever else they have a real home, traveling to London only to put in their time in a soulless business park. And drink. Lots.  Earlier I was listening to a woman’s conversation in what passes for a restaurant at the Heathrow Comfort Inn Hotel. At £14.99 a plate, I would think they’d be able to hire a competent chef. Instead, they serve tasteless slop, devoid of spices, though if you’re lucky, you’ll get a tough, chewy papadam with it. A papadam is crisp; it snaps apart. Not these. More than once I’ve watched people struggle to tear one to pieces before shoving a bit in their mouth, their eyes widening in gustatorial   horror. It’s symbolic of the entire menu.  Curiously, my browser says the word “papadam” is misspelled. In searching through their spelling suggestions, starting, perhaps not inappropriately, with the word “escapades,” I see the word “adumbrate.” What the hell is that? Looking it up, I see that it means, amongst other things, “to foreshadow.” What a fucking perfect word. I don’t have a spell checker, I have a coauthor. I will never look at Firefox the same way again (and perhaps will be more careful about what I type to whom).  Getting back to the woman, she was nattering on about this and about that and her voice had a quality like an itch under the skin, one where you quietly sit there and scratch and scratch and get no relief at all. That’s because she had an American accent. Finding American accents irritating is not something I expected upon moving over here, but then I remember I have an American accent, too.  Sitting in my bar a few days ago, I was reading a book, turning page after page, desperately trying to figure out how our hero was going to survive my next whisky, when I realized I couldn’t figure out what language the ladies at the table to my left were speaking. Well, one of the ladies, actually. Two of them spoke English, but the third was chattering away on her cell phone and, jarred away from my book, I wondered what country she was from.    I wondered what her childhood was like. I wondered how she, too, came to be in this overpriced hotel—whose incredibly friendly staff is its only saving grace—when the awful truth hit me: she was Scottish. My brain switched to the “Scottish” filter and I could slowly make out what she was saying. She was talking about picking up some food from the store, but with a thick, burly accent that would as soon stick a knife in your kidney as say, “‘ere on holiday?”  I usually don’t have too much difficulty understanding the thicker accents. A couple of weeks ago, enjoying a night out at the downstairs bar with an Irishman, a Glaswegian (someone from Glasgow, Scotland), and an English lady. We found ourselves well in our cups, teasing the Englishwoman because I, the American, had no problem following the conversation but she kept asking them to repeat themselves because their accents were too thick.  Later the conversation turned, as it naturally does, to quantum physics and cosmology. David, the Irishman, was trying to explain to the English lady something about the “shape of the universe”, but we couldn’t stop laughing because his Irish accent kept forcing him to say “the sheep of the universe”. Eventually, superstring theory became superwool theory and we decided the universe is filled with billions and billions of subatomic sheep. God is my shepherd. The Bible, the definitive text on biology, will, in a pinch, fill in as a physics textbook.  Last week, about a dozen of us went to see the movie 300 at the IMAX theater in Waterloo. If you’ve ever been to an IMAX theater, you know these are the largest movie screens in the world and they have to have special film for their projectors. These things are huge. Which is why you might understand why I say that the naked women were not exactly titillating. Nipples the size of my head are not erotic.  I might add that there’s not much point in seeing this movie. It’s gay porn without gay sex. And like most modern porn, it’s all action, no plot. After reading about the Battle of Thermopylae, upon which the graphic novel was based (which in turn inspired the movie), I was surprised to discover the movie wasn’t entirely historically inaccurate. That, or the screenwriter consulted Wikipedia before setting fingers to keyboard.  The movie, in a nutshell, is 300 very buff men wearing little more than red capes and leather codpieces the size of a small car, spraying gallons of blood everywhere while somehow remaining mysteriously free of it themselves, even their blades. Oh, and a little bit of sex in the beginning just to make sure that guys have everything they want out of a movie. I am dumber for having seen it.  On the way home, Little Miss Drunk Off Her Ass stumbled onto the Tube and, after being told that she was about to sit on something wet and unidentifiable in one of the seats, asked if she could sit down in the middle of all of us. Though there were still plenty of extra seats on the Tube and inviting yourself to join strangers is simply not done , we said that was fine and proceeded to have very strange conversation with her. She was older and might have looked posh were it not for the blood vessels on her face making it clear how she spent her free time. After a bit of conversation about the movie and the buff men, she looked around at all of us and said, with a habitual drunkard’s precise enunciation, “I do prefer my men a bit thinner.”  Which is why I don’t really feel bad for what I said next.  “So, what do you do besides drink?”  The look of shock on her face was magnificent. Those were glorious times.  ","title":"Gustatorial Adumbration and the Sheep of the Universe","url":"\/blog\/gustatorial-adumbration-and-the-sheep-of-the-universe.html"},{"body":"    Introduction  Caveats  Life (as we know it)  Aliens?  Organic Compounds  The Drake Equation  What Is Life?  Life as Biopresentation  Philosophical Objections to Biopresentation  Why is rethinking life important?  Searching for Life Off-World  Unlikely Candidates  Triton  Ceres  Europa  Ganymede  Interesting Candidates  Enceladus  Venus  Mars  Conclusion     Introduction  The question of whether or not we’re alone in the universe is a profound one. There is something terribly sad at the idea that our star is the only one that shines down on life. With a possible lower bound of $10^{22}$ stars in our universe    and a universe almost 14 billion years old, that’s an average of almost two million stars created every day . While estimates of the number of planets vary, we think there are probably at least as many “Earth-like” planets as there are stars, that seems like two million chances per day of life being created. Of course, it’s much more than that because for every day that a planet doesn’t have life evolve, it has plenty of additional days, years, eons for life to evolve.  Given the preceding, many people are convinced that life is plentiful in the universe, but so far, we’ve never detected extraterrestrial life amongst the stars. Are we alone? Maybe. However, we should look closer to home. In fact, there are many intriguing clues in our own solar system which are worth examining. I’ll cover some of them, but I’ll save the best for last because it’s very exciting, even if we did overlook it for decades.  First, I want a to give a some background information for those who are curious about the search for extraterrestrial life but haven’t dug into the topic, just how complex it is. If that’s less interesting to you, you can skip ahead to the best candidates for life in our solar system .  Caveats  As with any writing on this topic, everything here is going to be a gross oversimplification. Unlike many other writings on this topic, I’m going to try to give a broad (but not deep) background into many of the issues involved. Sometimes I’ll provide additional caveats that might distract from the text, so you can click on the \"   \" symbol for more context.  Life (as we know it)  First, when I write “life,” mentally add the words “as we know it” after that. I get tired of typing it. The definition of life is muddled, to put it kindly, and we only have one tree of life as a sample. We have no idea what alien lifeforms might be like.  Aliens?    Not gonna happen. Source    Always remember that if the question is aliens, the answer is “no.” This might seem surprising to some who’ve attended my talks and heard me warn about those who use words like “always” and “never” in describing complex topics. People who use those words are often zealots. I don’t think I’m a zealot, but the default position of those who understand science is that extraordinary claims require extraordinary evidence. The existence of extraterrestrial life would be one of the most extraordinary claims of all time. So far, there’s no evidence of aliens. Thus, your default position should be “no aliens.”    Organic Compounds  You often hear the term “organic compounds” or “organic molecules” in connection with the search for life. Be careful.  In chemistry, organic compounds are just chemical compounds that contain carbon-hydrogen covalent bonds. In the popular press, we tend to use the term to mean any compound containing carbon, but it’s widely misinterpreted as “molecules related to life.” They are often related to life, but “organic,” in this context, does not mean life.  The Drake Equation  Before we can search for life, we have to define it. But before we get there, let’s think about the Drake Equation  . This was created in the 1960s by the late Dr. Frank Drake as a thought experiment . Like “organic compounds,” this is also widely misunderstood. I’ve often seen it referred to as an equation that helps us understand the prevalence of life in the universe. In reality, it’s a tool to help us understand the possible number of currently communicating extraterrestrial civilizations that SETI  might be able to detect. For the equation, the number $N$ is the number of civilizations we might be able to detect. The equation looks like this:  $$N = R_{*} ~ \\times ~ f_{p} ~ \\times ~ n_{e} ~ \\times ~ f_{l} ~ \\times ~ f_{i} ~ \\times ~ f_{c} ~ \\times ~ L$$  Those terms are:   $R_∗$: the average rate of star formation in our galaxy  $f_p$: the fraction of those stars that have planets  $n_e$: the average number of planets that can potentially support life per star that has planets  $f_l$: the fraction of planets that could support life that actually develop life at some point  $f_i$: the fraction of planets with life that actually go on to develop intelligent life (civilizations)  $f_c$: the fraction of civilizations that develop a technology that releases detectable signs of their existence into space  $L$: the length of time for which such civilizations release detectable signals into space   Depending on how you fill in those numbers, we could be alone in the universe, or we could have many advanced extraterrestrial civilizations within our galaxy.  For our purposes, we’re most interested in $f_l$: how often does life evolve? In fact, we’re not even interested in just planets because life could possibly evolve on a moon (there are several interesting candidates). Some panspermia  advocates even claim that life could have evolved in space, though this isn’t taken seriously by most scientists.  To be more specific, we’re asking “what are the odds of life spontaneously originating from simple organic compounds?”   This is known as abiogenesis  and is the subject of much research. The current consensus on an answer is, “we have no fucking idea.” What’s particularly maddening about this is that we’re not likely to have a clue until we detect other life. If we don’t detect life, that doesn’t mean it’s not plentiful (absence of evidence is not evidence of absence).  But if detect a biosignature  or technosignature  of life on a star on the other side of our galaxy, that might only put a lower bound on the amount of life out there, not an upper. We’ll need multiple detections of life to get a good sense of its ubiquity. Nonetheless, detecting any other life outside our solar system suggests that it’s common throughout the universe.  So now it’s time to ask the hard question: What is life?  What Is Life?  Again, this must be prefaced with the “life as we know it” caveat. And I’ll come right out and say what’s on my mind: most definitions of life are a steaming pile of shit.  One definition of life  , as taken from Merriam-Webster, is “an organismic state characterized by capacity for metabolism, growth, reaction to stimuli, and reproduction.”  Right off the bat we see a problem. Mules, for example, don’t reproduce.   Would you say they’re not alive? Of course not.  And what about fire? It reacts to stimuli and grows  . We could even argue that it metabolizes what it consumes for energy to keep growing, but we don’t think of it as “alive.”  There are other properties of life, however. Homeostasis  is the body’s ability to internally regulate itself and keep functioning. Does fire exhibit homeostasis? It’s hard to argue that it does. However, warm-blooded and cold-blooded animals have completely different approaches to homeostasis and homeostasis evolved over time  , so there are different levels of homeostasis. Maybe we can argue that humans have a homeostasis level of ten while fire has a homeostasis level of one? A piece of flint, however, might have a homeostasis level of zero.  But why would we argue that?  Life as Biopresentation  When I was working on ETL  systems to reduce the cost of Phase III clinical trials  , I often found myself working with data supplied by many pharmaceutical companies. Company A might refer to “Dr. Robert Smith, St. Mary’s Hospital, Arlington, Virginia.” Company B might refer to “Dr. Bob Smith, St. May’s Hospital, Arlington, Virginia.” Are they the same person? “St. May’s Hospital” might be a typo. “Bob” is short for “Robert.” Maybe they are the same person.  In reality, it’s hard to definitively state “yes” or “no.” Instead, I had access to tons of information about each researcher, including email addresses, phone numbers, ages, and so on. I would assign scores (how reliable) and weights (how important) to each piece of data and if two researchers crosses a certain threshold, they were probably the same researcher.  This same approach should be taken to the definition of life. Instead of life being a binary “yes or no,” we should describe its “biopresentation,”   or “how present are the signs of life in a ‘thing.'” So for a virus:         Property  Score  Weight  Value      Metabolism  1  5  3    Homeostasis  0  4  0    Growth  0  2  0    Reproduction  3  4  12    Communication  1  5  5     In the above, a virus might have a biopresentation of 20. A human, however, would be radically different:         Property  Score  Weight  Value      Metabolism  5  5  25    Homeostasis  5  4  20    Growth  5  2  10    Reproduction  5  4  20    Communication  5  5  25     A human might have a biopresentation score of 100.    This would be particularly useful in describing abiogenesis. One hypothesis of abiogenesis is  :   Prebiotic atmosphere with ammonia, carbon dioxide and water vapor.  Lightning produces simple organic compounds that fall into solution in shallow water.  The compounds react further in a prebiotic broth, forming amino acids.  The amino acids link up with peptide bonds to form polypeptide chain proteins.  The proteins combine into more complex molecules that can replicate and metabolize simple substances.  Complex molecules and organic compounds form lipid membranes around themselves and start acting like living cells.   At all steps along the way, we could have something closer to what we call “life,” though the initial steps might have a biopresentation level of zero. Instead of life being a fixed point, it’s a gradual process. We don’t point to a thing and say “this is alive” because that gets us back to the arguments of what life is or isn’t. We simply describe its biopresentation.  This would be useful for biology today, too. Consider the endless arguments about whether or not viruses are live  . It’s often argued that viruses aren’t alive because they can’t metabolize outside of their host. That reduces metabolism to the primary (or sole) attribute of life.   But metabolism is the global term for anabolism and catabolism  . Catobolism is just breaking complex stuff into simpler stuff while anabolism is using that simpler stuff to do more complex stuff. Fire catabolizes, but doesn’t anabolize. Viruses do both. So maybe they’re alive? But some still reject that because, again, they can’t do that without a suitable host.  But what’s a host? Obligate parasites such as the protozoan Plasmodium   can’t survive outside of their hosts. Or consider humans. If I drop you into the middle of interstellar space without a space suit, you’ll stop metabolizing, too. So the context in which we can biopresent is also a spectrum. Viruses are starting to look more alive.  But wait! There’s more!  Most living beings have some ability to communicate with one another. Maybe all do. As it turns out, we’ve know for a few years that viruses can do this, too  . The human communication score might be at a “ten,” while viral communication might be a “one,” but fire’s communication is “zero.” The weight of this factor is probably pretty high, so fire’s biopresentation would be pretty low, but viruses could be easily crossing the threshold.  Pragmatist that I am, I want to skip the philosophical debates about whether or not something is alive and go with the pragmatic discussion of how closely something resembles life. However, we can’t ignore the philosophical implications.  Philosophical Objections to Biopresentation  At this point, I can see many biologists howling at the above. In a 2013 Wired article entitled It’s (Almost) Alive! Scientists Create a Near-Living Crystal  , the lede reads:   Three billion years after inanimate chemistry first became animate life, a newly synthesized laboratory compound is behaving in uncannily lifelike ways. The particles aren’t truly alive — but they’re not far off, either.   The crystals created had metabolism and mobility, but couldn’t reproduce. You can read the original study at The Royal Society website  .  For these crystals, one could argue that a), they’re not alive, or b), what their relative biopresentation level is. Some people would be uncomfortable describing crystals as alive, but if we’re to search for life elsewhere in the universe, are we going to argue that life which doesn’t share the same biochemical basis as our own isn’t alive?  Or consider self-replicating Von Neumann machine  . By the biopresentation argument, they would likely score very high, but are machines “alive”? Or apply biopresentation to The Gaia Hypothesis  . Is our planet alive? That’s definitely a philosophical question, but in light of biopresentation, depending on what factors to include, we could make a strong argument that it is.  Why is rethinking life important?  What does this have to do with searching for life off Earth? It’s simple. If we go with a binary “yes\/no” for life, it means that we have defined a limited, possibly incorrect, set of properties necessary to define life. Something not meeting those conditions might be disregarded. If, however, biopresentation is adopted, we can start defining “lifelike” properties and perhaps not be so quick to dismiss something that is otherwise alive.  If that seems stretched, consider the “Shadow Biosphere,” an idea that there may be another biosphere on Earth that we’ve not yet detected. The idea’s not popular, but it seems persistent. Here’s Fraser Cain, the founder and publisher of Universe Today, describing it.     Ultimately, if there is a shadow biosphere on Earth and we can’t detect it, this doesn’t bode well for detecting life off Earth.  Searching for Life Off-World  We can search for biosignatures in other star systems, but until we can visit   , it’s likely to be controversial. Technosignatures are more promising because a receiving a radio transmission of a sequence of prime numbers followed by an encoded message is pretty conclusive. But in the absence of technosignatures, we need to fall back on biosignatures and in our own solar system, we can definitely visit.  First, let’s remove a few unlikely candidates. They’re candidates because we can envision life being there. They’re unlikely because it may not be “life as we know it” and thus are less likely to recognize it as life. We might even exclude them for a practical reason: it’s hard to visit them with current technology and we’re unlikely to be able to detect life if we do so due to the limited equipment we can bring.  Unlikely Candidates  If you’re looking for information on extraterrestrial candidates for life in our solar system, they’re plentiful. I’m of the opinion that we should investigate all of them, given how hugely important I value the idea of finding a second (and third, and fourth, ...) abiogenesis, but as a practical matter, we need to consider the cost (biosignatures on Pluto wouldn’t necessarily justify the resources needed to launch a mission there) and whether we have detected biosignatures. When it comes to searching for life, low-cost targets with biosignatures must be our first priority.  Triton    Neptune’s moon, Triton Source    The moon Triton orbits the ice giant  Neptune. It’s the largest Neptunian moon and along with the Jovian moons Io and Europa, along with the Saturnian moons Titan and Enceladus, is one of the five geologically active moons in the solar system  .   Triton is fascinating because it orbits Neptune opposite of Neptune’s orbit (it’s the largest moon in the solar system to do so) and it might have an ocean underneath its ice covering. This suggests that ocean existed prior to it being captured  , which might make that ocean older than Earth. However, it takes years to get to Saturn  and the amount of fuel necessary (and the complexity of slowing down once you reach the planet) limit out ability to look for life. Given that the extreme cold suggests that life can’t exist, it’s probably a low priority for us.  Ceres  Ceres, orbiting in our asteroid belt, is the largest dwarf planet. It might have an underground saltwater ocean  . It might also have abundant organic compounds  . However, many things unrelated to life are “organic compounds,” so this isn’t particularly inspiring. If the ocean does exist, we don’t know how it came about and the data is still unconfirmed. Ceres is interesting and accessible, but as a candidate for life, it seems poor.    Europa  Yeah, I’ve seen the dive Europa  art. This moon of Jupiter is tiny, but it apparently has more water than all of Earth’s ocean’s combined. It almost certainly has an ocean beneath its ice crust  , but I rate it a poor candidate in the search for life because its constantly getting blasted by radiation from Jupiter and we’ve seen no biosignatures. The ingredients for life seem to be there, but I’d rather we focus our attention on worlds which are giving us hints something is there.  Of course, 2010: Odyssey Two  , by Arthur C. Clarke contained the warning “ALL THESE WORLDS ARE YOURS – EXCEPT EUROPA. ATTEMPT NO LANDING THERE.” I’m just the sort of guy who would be tempted to visit just because I’m told not to.  Ganymede  The Jovian moon Ganymede, the largest moon in our solar system, is particularly intriguing because it’s large, it has a subsurface ocean, it has a magnetic field to ward off Jovian radiation, and appears to have many of what we think are likely prerequisites for life. This article by astrophysicist Ethan Siegel lays out a case for investing Ganymede for life  . However, I’m unconvinced for the same reasons as Europa: no biosignatures have yet been detected.  Interesting Candidates  Enceladus    Saturn’s Moon, Enceladus Source    At first glance, Enceladus does not seem like a likely candidate for life as it’s a small, icy moon of Saturn. However, it’s geologically active, with cryovolcanoes (they spew water), a subsurface ocean, and probably hot vents in those oceans. More importantly, something is pumping out a lot of methane from that ocean  . Methane is considered a biosignature  because it’s usually generated by life (on Earth, of course) and tends to degrade quickly, meaning that a large amount of methane suggests that something is replenishing it.  The team that analyzed the methane data studied a geological process known as serpentinization  which could generate methane, but came to the conclusion that it would be impossible to generate as much methane as was seen. A sample return mission was proposed for Enceladus  , but sadly, NASA did not accept the proposal. Enceladus is a wonderful candidate for sample return because ...   Its low gravity makes the mission easier  Methane is a biosignature  The cryovolcanoes spew the samples high enough to easily collect on a flyby  The potential for discovering life makes this a very high reward mission   We’ll probably have to wait a few decades for the answer to this one, which is why I was tempted to put it in the “Unlikely Candidate” category (e.g., pragmatism).  Venus  Venus, despite having a hellish surface, has long been viewed as a candidate for extraterrestrial life. In 1967, Carl Sagan argued that life was unlikely on the surface of Venus  . However, in the same year, he argued that life might exist in the clouds of Venus  . We’ve also found dark streaks in the Venusian atmosphere that can’t be easily explained  . We have no idea what the dark streaks are. We don’t know their origin or composition. We don’t know why they don’t mix with the rest of the atmosphere and we don’t know why they absorb ultraviolet light. From the article:   However, in a region beginning around 31 miles (50 km) in altitude and extending 7.5 miles (12 km) outward is a sweet spot where the temperature ranges between 86 degrees F and 158 degrees F (30 degrees C and 70 degrees C), and the pressure is similar to that on Earth’s surface. Life could potentially survive in this zone where the dark-streaking UV absorber is found.   Microbial life could easily explain those dark streaks. Further, we now know that the unusual Venusian magnetic field is created by an interaction between the solar wind and Venus' ionosphere  . Venus' ionosphere starts about 120 kilometers above the ground, so we have a magnetic field to help protect potential life against cosmic rays. Without such a field, those rays could sterilize the planet.  But again, aliens? No. There’s probably a non-biological explanation (there always has been, once it’s found). However, until life is ruled out, Venus is relatively easy to visit (easier than Mars, actually), and if a sample return mission could be arranged, it’s worth the effort. Even if we didn’t find life, the scientific knowledge we would gain would be tremendous.  More exciting is the recent detection of phosphine in the atmosphere. Since phosphine is generally produced by life on Earth, and we struggle to describe any geological process on Venus that could produce phosphine, the news created a lot of excitement because phosphine is a considered to be an excellent candidate for a biosignature in anoxic (low oxygen) atmospheres  . In particular:   Phosphine is a promising biosignature gas, as it has no known abiotic false positives on terrestrial planets from any source that could generate the high fluxes required for detection.   Phosphine on Venus would be stunning news, but others have come out to say there’s no phosphine after all  , and others have disputed the original research  led by Professor of Astronomy Jean Greaves  and her team at Cardiff University in Wales. She’s had to post follow-up papers to explain flaws in some of the objections to her work  . The overall consensus in the popular press seems to be that there no phosphine detected in the Venus atmosphere and Greaves and her team simply misinterpreted the data.  If Greaves is right, this could be huge news. If she’s wrong, that’s how science works. However, in what should be a major story, but has largely been unreported in the press, The Planetary Society  has published a recent podcast with Professor Greaves  where she announces that another telescope has apparently confirmed the existence of phosphine  , meaning two telescopes and at least three different instruments confirming its existence. This is not proof (and we need more teams than just Greave’s team reporting this), but it certainly suggests we need to follow this up.  For more information, check out the Life on Venus  page on Wikipedia.  Mars    The planet Mars.   If I could only choose one place in the solar system to search for extraterrestrial life, it would be Mars, hands down. However, my reasoning is due to some research which, sadly, has been largely ignored. It was research conducted in the 1970s, but we’ll get back to that.  We now know that Mars was relatively like Earth early in its history, with a large ocean covering its northern hemisphere and a generally warm climate. However, due to the lack of a magnetosphere, Mars' atmosphere was stripped away by the solar wind  . With that, it lost the ability to trap heat and it turned into a cold, apparently lifeless rock. We don’t know the exact timing, but it may have been about 1 to 1.5 billion years ago that Mars lost its magnetosphere  , giving life plenty of time to evolve, given that life was well-established on Earth at this time.  However, that’s not all. Methane has possibly been detected on Mars, but the Trace Gas Orbiter did not find any methane  . However, some recent studies suggest that there may still be methane produced on Mars  by unknown sources. The existence of methane on Mars is still highly controversial, but I consider three things:   Mars was very Earth-like for billions of years  We might have detected a key biosignature  We are considering colonizing Mars   Those three things together make Mars a strong candidate for searching for life simply because we can do this as a side-effect of other ongoing efforts.  That is not, however, the primary reason we should choose Mars as our primary candidate in the search for life. Yes, we have a potential biosignature, but you might remember my biopresentation discussion above. We might have discovered metabolism on Mars, and that happened back in the 1970s.  From this phys.org article  :   In 1976, two Viking landers became the first US spacecraft from Earth to touch down on Mars. They took the first high-resolution images of the planet, surveyed the planet’s geographical features, and analyzed the geological composition of the atmosphere and surface. Perhaps most intriguingly, they also performed experiments that searched for signs of microbial life in Martian soil.   In the experiment, these two landers, thousands of kilometers apart, each injected a $C_{14}$-laced (carbon 14) nutrient solution and then examined the air. They each detected that something had released the $C_{14}$ from the solution. They also heated each sample of 50 degrees Celsius (122 degrees Fahrenheit) and repeated the experiment. Each detected significantly reduced amounts of $C_{14}$, as would be expected if the heat killed off organisms.  Previously, this experiment had been repeated extensively on Earth to verify that it was indeed a solid indication of life. It was also tested with known sterile samples to see if some other effect were causing the release of $C_{14}$.  So in 1976, we had a strong indicator of life, or “biopresenter,” as I like to think of it. However, there were some problems. The biggest obstacle could be if some natural chemical process could replicate these results. A type of salt called “formate” could possibly be responsible, but it’s not thought that formate is heavily present on Mars. Perchlorate, which we know is heavily present on Mars, can produce similar results, but it does so even when heated to above 160 degrees Celsius, so it’s not consistent with what we’ve seen. Hypochlorite might be present on Mars and it breaks down when heated to above 160 degrees, but its behavior has not been tested at 50 degrees,   . It’s currently the best explanation for the observed biopresentation, but even if the 50 degree test is carried out, that doesn’t rule out microbes on Mars. Of course, I remind you of the answer to the “aliens?” question. Read the article in full to understand more  . It’s a complex question and there are no clear answers.  Conclusion  For Mars, we had suitable conditions that life could have evolved in. Given the ubiquity of extremophiles on Earth, it’s not a stretch to imagine ancient Martian life evolving to exist in their current harsh climate. We have a potential biosignature and a potential biopresenter. Mars is our best bet for looking for extraterrestrial life.  However, there are so many other interesting questions here. Recent research bolsters the case for panspermia  , so if we discover life on Mars, did it originate on Earth? Or, since we know Martian meteorites have landed on Earth, are we Martians? I certainly hope not. I have nothing against the idea of of originating on Mars, but I want to find many examples of abiogenesis. That would tell us that our universe is probably filled with life. The more life there is, the more chances for intelligent life to exist somewhere else. Of course, that brings up the Fermi Paradox.  With the the new space industry taking off , it’s a very exciting time to be alive and exobiologists might turn from biologists who write science fiction to one of our most important scientific disciplines.    It’s Time.   ","title":"Searching for Extraterrestrial Life in Our Solar System","url":"\/blog\/searching-for-extraterrestrial-life-in-our-solar-system.html"},{"body":"  This blog entry is actually for my daughter. She’s eight years old and already turning into a math wizard. While other children are struggling with simple multiplication, she’s delving into square roots, negative numbers, factorials, and so on. She’s also been asking her teacher at school when they’re going to start getting problems in base 5 (she can count in different bases &lt; 10 now).  So when we were discussing factorials, she asked why the factorial of zero is one and I, um, stumbled. I was a math whiz in my high school, taking calculus in my jumior year, surrounded by seniors, but I’ve forgotten most of my math and for the life of me, I couldn’t remember the answer to her question. So here it is, after a bit of a refresher.  You probably remember that the factor of an integer, n , is the product of all integers from 1 to n . So we would say that that the factorial of 4, or 4!, is equal to:  $$4! = 4\\times3\\times2\\times1 = 24$$  And for a few other numbers:  $$3! = 3\\times2\\times1 = 6$$ $$2! = 2\\times1 = 2$$ $$1! = 1 = 1$$  But where does that leave the factorial of zero? There doesn’t appear to be any room left to put that equation.  There are two simple ways of answering this. One is intuitive and the other involves simple math.  The Intuitive Solution  Imagine that you have three runners in a race, Alice, Bob, and Charlie. Assuming ties are not allowed, how many ways can you arrange them into first, second, and third place? Mathematically, we’d ask “given a set of n elements, how many different ways can you order them?” That’s what’s called a “permutation.”  Another way of describing a permutation is “how many ways can you rearrange a set of items?”  Well, let’s work it out the hard way:   Alice, Bob, Charlie  Alice, Charlie, Bob  Bob, Alice, Charlie  Bob, Charlie, Alice  Charlie, Alice, Bob  Charlie, Bob, Alice   So for three participants, there are six different ways we can order them. Not coincidentally, that’s equal to 3! (three factorial). What if it’s only Alice and Bob?   Alice, Bob  Bob, Alice   That’s 2! (two factorial).  If you are really bored, you can work out four participants, but I promise you that it’s equal to 4! (four factorial).  So to figure out the number of ways you can order n elements, you just calculate n ! ( n factorial).  But what is n ? In math, we have the concept of a set .   A set is merely an collection of n distinct things. The order of those things doesn’t matter. The set {Alice, Bob, Charlie} is the same as the set {Charlie, Alice, Bob} . So our original question is “how many permutions of a set are there when n = 3?” And the answer is 3! (three factorial).  But it turns out that an important concept in math is the “empty set”, designated as {} . How many permutations does it have? Or to ask another way, “how many ways can we arrange nothing?” 1 and only 1. Thus, 0! = 1.  Do the Math  OK, that sounds like cheating, so let’s look at it another way.  As it turns out, we can say the following:  $$n! = \\frac{(n + 1)!}{n + 1}$$  How does that work? Well, let’s use 4! as an example.  $$4! = \\frac{(4 + 1)!}{4 + 1}$$ $$4! = \\frac{5!}{5}$$ $$4! = \\frac{5\\times4\\times3\\times2\\times1}{5}$$  And since the fives cancel out:  $$4! = 4\\times3\\times2\\times1$$  And working this out with other numbers:  $$3! = \\frac{(3 + 1)!}{3 + 1}$$ $$2! = \\frac{(2 + 1)!}{2 + 1}$$ $$1! = \\frac{(1 + 1)!}{1 + 1}$$ $$0! = \\frac{(0 + 1)!}{0 + 1}$$  Since it’s 0! that we’re interested in, we see it works out to:  $$0! = \\frac{1!}{1}$$  Since 1! is 1, we have:  $$0! = \\frac{1}{1}$$  And one over one is one. Thus, 0! = 1.  ","title":"Why is zero factorial equal to one?","url":"\/blog\/why-is-zero-factorial-equal-to-one.html"},{"body":"    Introduction  The Tyranny of the Rocket Equation  How to Get to Mars  How to Get to Mars, Extra Credit  The Physics of Perseverance  Rocket Engineering  Rocket Fuels  Nuclear Thermal Rockets  Max q and Bernoulli’s Principle  Re-entry and Hypersonic Flight  Conclusion     Introduction  The old space race was driven by governments, desperate to bolster national pride. The new space race, known in the industry as “NewSpace”, is driven by entrepreneurs, looking to make record profits. Science fiction is replete with stories of the horrors of corporate space colonies, charging workers for the very air they breathe and denying them basic rights. These seem far-fetched, but our history shows the dangers  . Even if well-intentioned, what happens when you’re living in a mining colony asteroid owned by Asteroids, Inc. and they go bankrupt?  There’s plenty to be concerned about, but nonetheless, we’re charging into this future and I confess, I love it. My sister, Gayle, taught me to read at a young age and while in the first grade, other students would be sounding out “see Spot run” while I was reading National Geographic. At home, my mother had stacks of old pulp magazines such as Astounding Magazine, The Magazine of Fantasy and Science Fiction, and so on. Ever since I can remember, I’ve dreamed of the stars and, despite the concerns, I’m delighted we’re finally taking the next steps to get there.  But how does it all work? Why is it so hard? What the hell is “max q”?  The Mars Society of Canada  has published a series of blog posts by Jin Sing Sia  about rocket physics. They’re introductory articles, but they’ll get you up to speed pretty quickly on the basics. You don’t even need to know advanced math.  There is, however, a slight problem with the articles. They didn’t include a table of contents! We don’t even have “previous” or “next” links for the articles. So allow me to do that, along with providing a summary for each. Note that these articles should probably be read in order, but it’s not strictly necessary.  The Tyranny of the Rocket Equation  link   If you follow space news, you may have heard about the tyranny of the rocket equation. In short, if you have a small amount of fuel, your thrust won’t last long enough for you to leave the planet. However, if you add more fuel, it’s not just lifting the rocket, it’s lifting the rest of the fuel! The more fuel you add, the harder it is to lift the rocket.  How to Get to Mars  link   Ever wonder why Mars only has “launch windows” every 26 months? What’s a “Hohmann transfer orbit”? Can we get to Mars faster than the projected nine months?  How to Get to Mars, Extra Credit  link   Discover the basics of orbital mechanics. They were touched on in the previous article, but this one goes a bit more in-depth. You can skip it if you want, but why would you want to?  Light physics and mathematics knowledge is recommended. Maybe you’ll finally learn what “perigee” and “apogee” are!  The Physics of Perseverance  link   By now, you know the Perseverance rover is cruising around Mars, taking selfies, doing amazing science and generally having a blast—up until the point when she realizes she doesn’t have a ride home.    The Perserverence Rover on Mars took a selfie.   The fact that Perseverance was able to land on Mars at all is amazing. The “sky crane” that lowered her to the surface was a thing of beauty. This article explains the approach, atmospheric entry, descent, and landing.  Rocket Engineering  link   Did you know that your car’s engine is more mechanically complicated than most rocket engines? On the other hand, your car’s engine doesn’t burn hot enough to melt the engine. Learn why rocket science ain’t easy.  Rocket Fuels  link   Liquid hydrogen is one of the best fuels for rockets, so why don’t more rockets use it? What is hydrazine? Why do we use hypergolic fuels when they’re so dangerous? Wait, what’s a hypergolic fuel?  Nuclear Thermal Rockets  link   Nuclear rocket engines are amazingly powerful. Their exhaust also tends to be amazingly radioactive. And if a launch goes poorly, you can’t just use your FTS (Flight Termination System) to blow it into little pieces lest you shower everyone with radioactive waste. So why are we considering them?  Max q and Bernoulli’s Principle  link   Max q and Bernoulli’s Principle are two terms you often hear when space nuts discuss rockets. Understanding them helps to explain, amongst other things, why the opening scene to the otherwise excellent film (and book), “The Martian”, was so unrealistic.     Re-entry and Hypersonic Flight  link   You know space capsules have heat shields to protect them from the heat of reentry to our atmosphere, right? It’s not because of friction.  You’ll also learn why sound travels much faster under water.  Conclusion  Space is amazing. And it’s hard. I hope you enjoyed reading through all of the linked articles.  The NewSpace race is well under way and we’re finally going to get to the planets, even if we don’t colonize them. I am hopeful, however, that eventually we will.  ","title":"Rocket Physics 101","url":"\/blog\/rocket-physics-101.html"},{"body":"","title":"Why Black People Hate Your Dreads","url":"\/blog\/why-black-people-hare-your-dreads.html"},{"body":"  Defund The Police?  The phrase “Defund the Police” has flowed across the US like a tidal wave, wrecking all conversations in its path. In the wake of the murder of George Floyd , public anger has reached a point where it’s tearing the US apart. While it would be too simplistic to claim that police brutality is primary driver beyond the discontent in America today, it cannot be ignored. However, “defund the police” has to be one of the most idiotic rallying cries people have rallied behind.    Police in riot gear. Source    This is largely because defunding the police isn’t what the protests are trying to accomplish  . Instead, the movement is about allowing police to be police, dealing with crime, and not having SWAT teams serving eviction notices  . So if a homeless guy urinates on a sidewalk, trained social workers can evaluate the person, determine if there’s an actual threat, and assist them in getting help, along with gaining access to appropriate services.  Or just send out men with guns who receive little training in social issues and who will toss the homeless person in jail. I’m not sure how that helps anyone. Urinating in public puts you at risk of a year in jail (two in Ohio) and being placed on the sex offenders registry  . Is this a proportionate response? Is destroying someone’s life a healthy response when the alternative is to offer to help the person get back on their feet?  But instead of a slogan such as “let police be police” (in my reading of police forums online, many police support this), many are calling for defunding the police, which those on the other side disagree with because they think it means “eliminate the police,” despite many people trying to clarify the meaning. As anyone in marketing can tell you, if you have to explain what you meant over and over again, then clearly what you said communicates something different from what you meant.  But what‘s the alternative? All but the most authoritarian amongst us admit that there is a problem; it’s the solution we disagree on. Is there a better way?  I’ve been reading through the blog of Donald DuPay  , a former Portland police officer. In particular, he wrote Bad Training and a Culture of Lying Continue to Haunt PPB  . While much of the language is strong (for example, the “n” word is sometimes used, but in a clear context), if his account is true, he lays out a strong case that there are serious issues with policing. In my reading of police forums, I can only attest to this. What DuPay does, however, is interesting. In particular, he cites the 14th Amendment of the US Constitution  , of which Section 1 reads, in part:   [No state shall] deny to any person within its jurisdiction the equal protection of the laws.   Internal Affairs  What is “equal protection”? As DuPay argues:   It does not state for instance, that electricians may have their own private justice system. It does not state that plumbers may have their own justice system. And it does not state that the police can have their own private justice system either, and Internal Affairs is a private justice system for the police, which make it unconstitutional.   DuPay cites many instances of Internal Affairs (IA) investigating criminal behavior of police officers and giving them a slap on the wrist. Or at the very least, giving them punishments far less severe than what the public would face, but still allowing the offending officers to keep their jobs. Officer DuPay cites instances of officers being investigated by IA and being allowed to keep their jobs, despite committing theft or drunk driving, and further claims that:   Over 80 percent of cases that go before IA are whitewashed away and swept under the proverbial rug, with officers either not being charged or given flimsy disciplinary actions for serious infractions involving police misconduct.   And what would the alternative be? Simple: let the police go through the same justice system anyone else has to go through. If they’re found not guilty, let their union or jurisdiction pay for their defense. If they’re found guilty, they pay for their own defense and they suffer the same penalties non-officers would face, with the possibility of greater penalties if they’re found to violate Section 242 of Title 18  which   makes it a crime for a person acting under color of any law to willfully deprive a person of a right or privilege protected by the Constitution or laws of the United States.   Of course, there are problems with this. Sometimes a police officer might engage in illegal activity as part of undercover work  . For example, a police officer might receive stolen goods and later fence them in order to break apart a criminal organization. The buying and selling of drugs is another example: if you can’t show the criminals have willingly engaged in the activity ( assuming you are not engaged in entrapment  ), then it can be very hard to secure a conviction.  Or maybe the case against the officer requires exposing a confidential informant whose life may then be jeopardized? Again, this one is difficult because when laws are being broken, questions of right and wrong can become murky. However, Human Rights Watch has researched IA divisions and found  :   Internal affairs divisions must be central to any examination of how police departments deal with abusive behavior by officers. Therefore, it is alarming that no outside review, including our own, has found the operations of internal affairs divisions satisfactory. In each city we examined, internal affairs units too often conducted substandard investigations, sustained few allegations of excessive force, and failed to identify and punish officers against whom repeated complaints had been filed. Rather, they, in practice, often shielded officers who committed human rights violations from exposure and guaranteed them immunity from disciplinary sanctions or criminal prosecution.   And honestly, this is probably a kindness to many IA officers. Typically they don’t spend their entire career in IA. Instead, they were once regular police officers and they’re now called on to investigate their friends and colleagues. The conflict of interest is clear and it must be excruciating. Just consider the well-known problem of steroid abuse by police  . They have a very dangerous job and if they are afraid to to subdue a violent suspect, they may do what many police officers are doing: use steroids and hit the gym. And you, as an IA officer, formerly a beat cop, may have tremendous sympathy for them. And you also know you won’t be an IA officer forever and will be working with other police officers again. For many, this can be an impossible situation.     What I find appealing about this solution is that the IA officers don’t lose their jobs: they simply revert to be regular police officers. Further, it leverages existing systems rather than requiring the creation of new ones. Additional safeguards will need to be in place to account for the unusual nature of police work, but why should an officer who passed out drunk, on duty, in their car, get to keep their job, much less escape the criminal sanction everyone else would face  ?  I “get” the motivation behind “Defund the Police.” More than that, I completely support it. But there comes a moment when we need to distinguish behind what we should do and what we can do. Maybe getting rid of Internal Affairs and having the police face the same justice system as everyone else is an effective compromise. It won’t fix the problems, but it could be a start.  ","title":"Don't Defund The Police","url":"\/blog\/dont-defund-the-police.html"},{"body":"  It’s a testament to my late father that he managed, even in death, to surprise me. While recently in Germany for Jim’s funeral, I discovered that Jim admitted to my sisters that yes, we probably have a sibling or two in Russia. Given Jim’s propensity for mendacity when discussing his children (in particular, denying their existence), his admission was a surprise.    Jim’s ashes.   Just to recap for those not following the story: our father, Jim, had—that we know of—seven children by four women. Aside from children who had the same mother, we didn’t know about each other and it was many years of work to find out about each other’s existence and eventually locate each other. We’ve grown very close, despite our father, and now it turns out there may be “one or two” more, probably born near Moscow in the early 70s. That’s not much to go on.  So what do we have to go on?  KGB  I’ve been told that the KGB would have had, at the very least, cursory files on any Americans living in the Soviet Union during the height of the cold war. Given Jim’s background and the fact that he spoke fluent Russian, he would clearly be a person of interest. The KGB files might have information about Soviets that Jim knew (“knew” being such an overloaded word there). Though I’ve a few Russian contacts, I assume that any such records are not available to me.  Janath Poe  While in the USSR, Jim was married to woman named “Janath Valentine Poe”, possible family name of “Heritage”, probably British. I haven’t found her. She might have more information. (I’ve found hints of her online, but I don’t know if it’s the right person and I don’t have contact info).  Dennis Blewitt, Daily Mail  One of Jim’s friends was Dennis Blewitt, a journalist working for the Daily Mail. Blewitt was later expelled from the USSR in 1985  in apparent retaliation for the UK expelling Soviets accused of espionage. I’ve reached out to the Daily Mail a few weeks ago, but have not heard back from them. I was hoping they could put me in touch with Mr. Blewitt and that he might have some recollection of who Jim knew.  The Letter  In Jim’s effects I found a letter from Sweden written to Jim in 1978. I shared that and some other information on Reddit and a sharp-eyed reader  managed to make out a nearly illegible signature and produced the names “William Aldridge” and “Belkacem Bazi”. These two aspiring filmmakers went on to make a number of films together. Given some clues in the letter, I found that the writer, Aldridge, likely knew Jim when he lived in the Soviet Union. Aldridge wrote:   In any case, I have no forgotten my debt to you, nor has Belkacem who sends you his regards.   Sadly, further research has revealed that both of these men have passed away. Short of me finding family members who know something, I’ll never learn exactly what assistance Jim offered to these two men. At the time of the letter, Aldridge was in Sweden and Belkasem was in Canada. There’s no suggestion of a connection with the Soviet Union, though in 1999, these two men collaborated on a French series about Russian cinema.     My sisters, Lynne and Gayle, sitting at the Idsteiner Brauhaus, in Idstein, Germany. It was Jim’s favorite watering hole.   So that’s where we’re at. Aside from Janath Poe and Dennis Blewitt, we have no leads. Jim, to the very end, provided no help. Going through his documents has also provided little help, but we’ll press on. We have more siblings to discover!  ","title":"Missing Siblings?","url":"\/blog\/missing-siblings.html"},{"body":"  The Stupid Experiment  I started a “Facebook experiment” after one of my posts went viral with millions of views. It was a lovely video of excited dogs at a shelter joyfully picking out their own Christmas toys.     With this temporary internet fame, I decided to expand my circle, to engage with those I would not ordinarily engage with, to respectfully counter opinion with facts (and references), and to try to understand those who have differing points of view. In short, I was deluged with friend requests and I accepted almost all of them.  I can now share the results of that experiment: “People on Facebook are terrible.”  There are exceptions, but generally, the pattern has been this:   Someone posts something easily disproved  I politely inform them that they may have encountered “fake news”  I provided links (trying to use sources they would view as credible) to give them more information   The result, in almost every case, is that I was ignored and they continued posting fake news.  In a few cases, abuse was hurled at me. For politely providing correct information, I was called an “idiot”, a “troll”, a “socialist”, and, one one memorable occasion, a “Nazi on the payroll of George Soros.” Given that Soros is a Jew who survived Nazi-occupied Hungary, the last was particularly sad.  And once, I received a polite “thank you” and once a post was taken down in response.  Very few people were willing to engage constructively with me. Only one provided references to back up their point of view. I thanked them for that.  In short, the experiment was a failure. Facebook is clearly a platform that people use to vent, not to learn. And frankly, the discussion was depressing. So I’ve stopped.  I’ve started to unfriend those who are particularly bad. Generally this was easy. Obvious bots and those posting “Obama is a Muslim” memes are easy to discard.  But one was an old friend who kept posting blatantly false anti-vaxx information. It was trivial to verify it was wrong, but they claim that they—unlike me—had researched the topic for years. I finally had enough when they were sharing anti-vaxx propaganda that was to “protect the children.”  Another was an old friend who was basically being an apologist for systemic racism. They posted a few quotes from far-right black people in an attempt to justify their viewpoint that there is no systemic racism today, aside from “affirmative action programs”. I’m not going to engage with people like that. It’s too much for me. Racism, whether it’s overt or stemming from a willful disregard for what’s happening in society, is a line in the sand for me.  So I will continue to prune my friend’s list back. Some say that I’m just “staying in my bubble.” That would be true if Facebook was truly a forum for engagement with those who have different points of view. As I’ve found, it’s not.  The Reality  None of the above is surprising to anyone who follows social media: there has been a massive breakdown in social\/political discourse over the past decade. Social psychologist Jonathan Haidt  has spent years researching this and uncovering the root causes. In his article, Why the Past 10 Years of American Life Have Been Uniquely Stupid  , Haidt breaks down the process by which we have grown increasingly polarized. This is largely due to “innovations” in social media. Via “sharing” behavior, coupled with “likes,” we’ve created a system whereby those posts providing provocative content, particularly context which provokes anger, tends to get shared more frequently. People get angry, critical thinking flies out the window, and another angry rant gets a like and a share. We condemn our enemies, flaunt our virtue, and the world gets just a tiny bit darker.  This has been happening millions of times a day, every day, for over a decade and it’s fracturing out society. Truly this is a death by a thousand cuts.  I fully confess to taking part in this, though I’m trying to break out of this cycle. More and more there’s talk of functioning democracy breaking down in the US and perhaps even the US itself breaking apart. Lest that sound alarmist, allow me to quite from the 2022 Texas Republican Party platform  (emphasis mine):   Pursuant to Article 1, Section 1, of the Texas Constitution, the federal government has impaired our right of local self-government. Therefore, federally mandated legislation that infringes upon the 10th Amendment rights of Texas should be ignored, opposed, refused, and nullified. Texas retains the right to secede from the United States , and the Texas Legislature should be called upon to pass a referendum consistent thereto.   Equally ominous:   We reject the certified results of the 2020 Presidential election, and we hold that acting President Joseph Robinette Biden Jr. was not legitimately elected by the people of the United States.   But it’s not just Republicans! 47% of western Democrats have expressed support for breaking away and forming a new country  .  How Do We Fix This?  Enter Jonathan Haidt again. In a 2008 Ted Talk  , Haidt shares the results of his research regarding the moral roots of liberal and conservative thought. He’s found these roots to be shared across the world. The five key roots are:   Ensuring fairness  Caring and compassion  Respecting purity  Respect for authority  Loyalty to your group   Interestingly, while liberals and conservatives generally agree on the first three, liberals tend to reject the last two. Trust in authority must be earned and if the authority abuses that trust, it must be taken away. Group loyalty is also problematic. “America, right or wrong,” is not a liberal ideal.     The conservatives counter, correctly, that while it’s imperfect, authority and loyalty are key to building and maintaining society. Our society works .  The liberal response? It works for you . There are many minority groups in the United States who still do not have equal access to education, jobs, or legal process. Conservatives would prefer that any changes be made slowly, if at all, lest we threaten the fabric of society. Many liberals would push changes through quickly, arguing that we must become a more just society.  So who’s right? Which side should win?  Neither, because thinking in terms of winning or losing is framing the problem incorrectly. Conservatives are correct that authority and loyalty help to buttress society. Liberals are correct that we must evolve society. I tend to be sympathetic to the latter point of view, while acknowledging that it does not always work.  We cannot simply destroy existing institutions without offering credible alternatives. For example, in the United States, much public education is funding via local property, income, and sales taxes. If you live in a poor area, your school is likely to be poorly funded and your education will suffer as a result. We know this is a problem, so how do we fix this?  Well, it turns out we are actively working on solving this, though different states have significantly different approaches, often for very good reasons. In particular, much work is being done to target poor school districts with extra money  . However, that doesn’t actually solve the issue of poverty itself. Simply growing up poor hurts your education outcomes  , starting at kindergarten. Worse, poor children are much more likely to drop out of school, either due to financial or social pressures. We want to fix this, but the fixes themselves are not always clear.  Or take the idea of defunding police . In reality, despite the terrible choice of name, the core idea isn’t fundamentally unsound, but how it’s been viewed (and sometimes promoted) has been terrible. Cut back their budgets substantially to «insert one of numerous explanations here». So if the police forces are smaller and can respond to fewer incidents, what happens when incidents aren’t responded to?  A more reasonable approach is to let the police focus on actual crime and allow social services to handle actual social issues. A mentally-ill homeless man peeing in a doorway isn’t going to get better by being swarmed by gun-toting uniformed officers and then spending time in jail. I have often read police boards where officers lament that they became a police officer to enforce the law, not to be ad-hoc marriage or drug counselors.  If we find a drug addict, stumbling around and acting erratically, do we get them medical help? Do we send them to jail? Do we do both? Should social services respond first, calling the police if they need help? Or should police respond first, ensuring public safety, and then call social services? And if a person clearly needs help, do they lose that right simply because they engaged in the illegal behavior of taking drugs?  Probably everyone one of those answers might be appropriate, depending on the particular situation, but do we have the twin luxuries of money and time to decide every incident on a case-by-case basis?  Instead of conservatives and liberals trying to defeat each other, the sides should look to the unique strengths of the others and cooperate. Liberals should continue to push forward to make the world a better place, while conservatives should be calling them out if their proposals are ill-thought. For example, the state of Oregon recently decriminalized possession of small amounts of cocaine, heroin, LSD, methamphetamine, and other drugs  , but didn’t put the resources in place to provide medical assistance to those in need. Those resources were funded , but the infrastructure didn’t simply didn’t exist, nor were clearly targeted goals outlined. While several countries (most notably, The Netherlands and Portugal) have had very successful decriminalization programs, Oregon is reporting increases in addiction and overdoses  as understaffed hospitals struggle to keep up with the demand. I expect (hope?) they’ll fix this over time, but idealism without pragmatism is a dangerous thing.  The US has the highest incarceration rate in the world  , with almost 1% of the adult population behind bars. As of this writing, there have been 322 mass shootings in the US  , almost two a day, and the number is on the increase. White nationalism is on the rise  , with a 39% increase in hate crimes large cities in 2021  . Teen mental health has declined dramatically across the world  , with much of this being tied to social media consumption.  If we can’t do this for ourselves, can we do this for our children? We need to find the strength to stop sharing negative social media. We need to put forward fact-based information that can be independently verified. If something angers you and you feel “triggered,” perhaps take a break? Just as important, ask yourself if it’s really true. And let’s try to find a way to point fewer fingers, and read the points of view of those you disagree with, especially if they offer solid information to support their point of view.  I don’t know how society is going to be past this. We’re in trouble. We need to stop electing politicians who are little more than walking memes, but I don’t see that happening any time soon. I worry that we won’t be able to put things back together unless they break first.  ","title":"My Facebook Experiment","url":"\/blog\/my-facebook-experiment.html"},{"body":" If you've been in this business long enough, sooner or later you're going to be faced with a terrible problem: fixing the legacy codebase. What follows isn't the only way to proceed, but it's a tried-and-true strategy that unfortunately doesn't appear to be well-known. The core of what follows is risk minimization. Assuming you're facing the problem of fixing a legacy application, you already have risk and you don't need more added to it. What follows is lower risk and lower cost than rewriting the system from scratch.  If you like what you read here and you need help, especially with Perl, get in touch with me and see how our company can help out.  Why You (Probably) Don't Rewrite The Code  Before we start, there are a few things you should know. First, read this now-famous Joel Spolsky article about why you should never rewrite your code (trust me, read it, but don't forget to come back). In that article, Spolsky makes a strong case about why you should refactor your codebase instead of rewriting it. Refactoring, if you're not familiar with the term, is the process of making a series of gradual improvements to code quality without changing the behavior. When you're trying to fix code, trying to change its structure and behavior at the same time is begging for trouble.  That being said, I don't believe in the word \"never\". If your code is written in UniBasic, rewriting might be your only option since you can't find developers who know the language (or are willing to learn it). Heck, I used to program in UniBasic and I've forgotten the language entirely.  Or if you're working with a relatively small piece of software with low impact, rewriting may not be that dangerous.  But let's say you can find or train developers for the language your software is written in, the software is mission-critical, and it's a very large codebase. Rewriting begins to make much less sense. Refactoring it means that you always have working code, you're not throwing away business knowledge or obscure bug-fixes, and your developers aren't starting from scratch, hoping they can make something work. In other words, you're minimizing your risk.  That being said, many companies (and developers) still opt for the rewrite. New code is exciting. New code promises new opportunities. New code is fun but fixing old code is often seen as drudgery. However, if you have a large, legacy codebase, the new code you're writing is, by definition, a large project and large projects are very high risk (emphasis mine):  In a landmark 1995 study, the Standish Group established that only about 17% of IT projects could be considered \"fully successful,\" another 52% were \"challenged\" (they didn't meet budget, quality or time goals) and 30% were \"impaired or failed.\" In a recent update of that study conducted for ComputerWorld, Standish examined 3,555 IT projects between 2003 and 2012 that had labor costs of at least $10 million and found that only 6.4% of [IT projects] were successful .  That's an old study, but there's still plenty of newer work which bears this out. The larger the project, the larger the risk. In fact, of the large projects I have been involved with for various companies, few were both on time and on budget. Some were cancelled outright and still others dragged on, long after it was clear they were a disaster, simply because no one wanted to take the blame for failure. One of them today is approaching its fourth year of a one-year schedule and is riddled with bugs and design flaws, but the company made the new software backwards-incompatible, switched over their clients and now have no way out. The only reason the company is still in business is that they bought another company that is very profitable and is paying for the company's mistake.  That last examples alludes to a dirty little secret that's often not talked about in our industry: large-scale rewrites often exchange one pile of spaghetti code for another. Rather than truly solve the underlying problems, the companies have traded a known set of problems for an unknown set of problems. If you need to fix your legacy code it's because you need to minimize your risk; why on earth would you knowingly adopt unquantifiable risk?  How to Refactor Your Legacy Code  Assuming you've decided that you don't want to face the cost and risk of a large-scale rewrite, how do you refactor your code?  First, you need to assess where you are. At a bare minimum:   What are the functional requirements of the code? (very high-level here)  What documentation exists, if any?  What areas of the code are more fragile? (Check your bug tracker)  What external resources does it require?  What tests exist, if any?   All of these things need to be written down so that anyone can consult this information at a glance. This information represents the bare necessities for the expert you're going to hire to fix the mess. If the above list seems simplistic, that's because we're refactoring, not rewriting.  And yes, you're probably going to hire an outside expert. Not only will they see things that you didn't, but while your current developers may be good, if they can't clearly lay down a solid plan to fix the legacy codebase while simultaneously minimizing risk, you need to bring in someone with experience with this area. What follows is not always intuitive and the expert's experience will help you navigate the rough waters you're already in. At a minimum, your expert needs to have the following:   Expert in the primary language(s) of your codebase  A strong automated testing background  Very comfortable with code coverage tools  A strong database background (probably)  An expert in system design\/architecture  Ability to admit when they're wrong  Understanding of business needs  A persuasive personality   The last points seems strange, but hard choices will need to be made and there will be strong disagreements about how to make them. It's hard to find this mix in top-notch developers, but it will definitely pay off.  Getting Started  The first thing you'll want to do is get a rough idea of how you want your new application laid out. Call this your architecture roadmap, but keep in mind that your landscape will change over time and this roadmap should be flexible. This is where your expert's architecture skills will come in. Various functional parts of your application will be decoupled and put into separate areas to ensure that each part of your application has a \"specialty\" that it focuses on. When each part of your application has one area it focuses on, it's easier to maintain, extend, and reuse, and that's primarily why we want to fix our legacy codebase. However, don't make detailed plans at this time; no battle plan survives first contact with the enemy. Instead, just ensure that you have a rough sense of where you're going to go.  Next, you're going to refactor your application the same way you eat an elephant: one bite (byte?) at a time. You'll pick a small initial target to get familiar with your new tools. Over time, it will get easier, but you don't want to bite off too big a chunk when you get started.  Refactoring a large application means writing tests, but unless you know what you're doing, you're probably going to get it wrong. There's often little TDD here — the code is already written — and you can't write tests for everything — you'll never finish. Instead, you'll be tactically applying integration tests piece by piece.  The first thing you need to do is understand what won't change in your application. By \"won't change\" I mean whatever it is that uses your application's output, whether it be through a JSON API, a Web site, a SOAP interface or what have you. Since something has to use the software, that something is what is going to make everything work. You're going to be writing integration tests against whatever that something is. For the sake of argument, we'll assume we're refactoring a Web application. You've decided that you'll start by writing tests to verify that you can list users on your admin page.  Inside those tests, you'll create a browser object, log in as an admin user, fetch the users page and write tests to assert that the expected users show up on that page. Just getting to this point can often take a huge amount of work. For example, how do you get code to connect to a test database? How do you ensure data isolation between tests (in other words, the order in which tests are run should not matter)? Heck, how do you create that browser object (hint: Selenium is a good choice here)? These and many more questions need to be answered when you're first starting out.  Getting to to this point may be easy if you already have some tests, or it may be very hard if you don't, but it's the important first step in the refactoring.  Once you have that first integration test targeting a small and (relatively) unchanging part of your interface, run your code coverage tools over the test(s) to see what code is covered with these high-level integration tests. Code which is covered is code which is generally safe to refactor (there are plenty of exceptions, but that's another article entirely).  Now you can start looking at which functional parts of the application are embedded in that tested code and make a plan for moving those sections into your architecture roadmap. At this point, it's tempting to rip everything apart, but don't give in to that temptation. Instead, focus on one piece at a time. For example, if you have SQL scattered throughout the code, start pulling that out into your architecture roadmap so that you have a clean API to work with the data you need. Or perhaps you have a Web application and you have been printing the HTML directly: look at using a templating system and start pulling the HTML out into templates. Don't fix everything at once or you'll be trying to do too much. Instead, focus on one area of responsibility and understand it well.  Don't Do Unit Testing (Yet)  Note that we've been talking about integration testing but not unit testing. There's a very good reason for that: with heavy refactoring of a legacy system, your units will change quite heavily when you first start, but the integration tests focusing on the rather static interfaces will not. You want to spend time refactoring your application, not your tests, so until you've stabilized how the code works internally, unit tests can actually be a distraction. Integration testing has the advantage that you can cover (if not actually test) huge portions of your code at once and if done correctly, can be very fast to write. Further, with poorly structured applications, unit testing may be very difficult, if not impossible. Integration testing will also help uncover bugs that unit testing cannot: bugs where different components have different expectations when talking to one another. However, there are some downsides to integration testing:   Integration tests run slower than unit tests  Bugs are harder to track down  It's easier to break real things if you've not isolated your code well enough   That being said, the advantage of integration testing at this stage is clear: refactoring is much easier when you have some basic tests to protect against the worst errors. It's also worth keeping in mind that if you've done little to no testing before this, you're not significantly worse off if you have some solid tests than if you have none. Don't obsess too much on this point: you don't want perfect to be the enemy of the good.  If you haven't already implemented a continuous integration (CI) system, this is the time to start. Even if your developers forget to run the tests, your CI system shouldn't. You want to find out fast if tests are failing.  Pushing Forward  After you've started refactoring one functional piece of a small part of your system, you'll probably quickly uncover some bad assumptions made in the original plan. That's OK. You've started small to minimize your risk. Correct those bad assumptions and then start integration tests with code coverage for another small part of your system, pulling out the functional portions (database calls, HTML, or whatever) that you've already been working on. When you feel comfortable that you've shaken out some of the worst issues, start looking at another functional bit of the system that your currently tested code shares and see if you can pull that out.  Note that this is where your expert's architectural skills are going to shine. They'll understand the importance of decoupling different functional portions of the application. They'll understand how to write robust, flexible interfaces. They'll learn to recognize patterns in your business logic which can be abstracted out. Do not hand this responsibility over to an existing programmer unless you are absolutely confident they have the skills and experience necessary to get this done.  At this point, what follows is a wash\/rinse\/repeat cycle which in worst case scenarios can take years to finish. It takes a long time, but it has some significant advantages:   The code is always working  You're not paying to maintain two systems at the same time  Business knowledge is not lost  New features can still be added  Tests can now be easily written to target existing bugs (even if you don't refactor that code yet)  You can always stop if you've made your codebase \"good enough\"   Why does this approach work? Any large project can seem daunting, but by breaking it down into smaller, manageable pieces, you can at least know where to start and get a sense of where you are going without the nail-biting worry about whether or not a huge project is going to fail.  When I've used this technique before, I've often found that it's a pleasure to finally have a cleaner sense of how the code is evolving and the current experienced team doesn't face the demoralizing prospect of watching their jobs disappear. The downside of this technique is that while code quality improves tremendously, there's always a feeling that it's not good enough. However, as I previously alluded to, many rewritten systems merely create new design flaws to replace old ones. This is far too common of a problem and it means swapping known problems for unknown ones.  Conclusion  The above strategy isn't appealing to many people and it can be a hard sell to those who are convinced that newer is better. In fact, in many respects it can be viewed as boring (though I love refactoring code), but I've successfully used this approach on multiple legacy codebases. However, if you're still trying to decide between a rewrite and a refactor, keep in mind that this approach is a relatively low-cost, low-risk approach. If it proves unworkable, you've likely risked very little. If the rewrite proves unworkable, you could cost the company a ton of money.  So the final question to ask yourself is when you should consider fixing your legacy codebase. The only advice I can offer is to suggest that you not wait until the storm before you fix your roof. Fixing a legacy code base isn't rocket science, but it does require a degree of expert knowledge in how to transform an existing codebase. Sadly, it's not a skill most developers seem interested in acquiring, but then, most don't seem interested in working on legacy codebases in the first place.  ","title":"A Simple Way to Fix Legacy Code","url":"\/articles\/a-simple-way-to-fix-legacy-code.html"},{"body":"    Introduction  Less Boilerplate  param and field  Attribute Shortcuts  Cloning  Customization  Conclusion     Introduction  If you’re waiting for the Corinna OOP project  to land in the Perl core, you’ll have to wait. Paul Evans  will start working on the implementation after Perl version 5.36 is released. However, it’s a lot of work and it’s not clear that if it will be ready by 5.38 and even then, it will be experimental. You’ve a couple of years to wait.  To bridge the gap, I’ve released MooseX::Extended  . It’s not really “Corinna-lite” because the way Moose  works and the way Corinna works aren’t quite compatible, but it will make your Moose code easier to write.  Less Boilerplate  The bane of many programmer’s existence is boilerplate. We hate boilerplate. Not only do we have to type the boilerplate, but we also have to remember it. For a simple, “read only” 2D point class in Moose, the class and my boilerplate looks something like this:  package My::Point::Moose; use v5.20.0; use Moose; use MooseX::StrictConstructor; use Types::Standard qw(Num); use feature qw( signatures postderef postderef_qq ); no warnings qw( experimental::signatures experimental::postderef ); use namespace::autoclean; use mro 'c3'; has [ 'x', 'y' ] =&gt; ( is =&gt; 'ro', isa =&gt; Num ); __PACKAGE__-&gt;meta-&gt;make_immutable; 1;  That’s a lot of boilerplate. Some of it’s not needed for this module, so I am often lazy and don’t write the boilerplate I don’t need, meaning I have to go back and add it later as the class grows.  Here’s the same code with MooseX::Extended :  package My::Point; use MooseX::Extended types =&gt; [qw\/Num\/]; param [ 'x', 'y' ] =&gt; ( isa =&gt; Num );  That’s right, you don’t have to declare it immutable or end it with a true value. MooseX::Extended handles that for you (though you can disable this).  Hmm, what if I want to subclass that and make it mutable?  package My::Point::Mutable::Moose; use v5.20.0; use Moose; extends 'My::Point::Moose'; use MooseX::StrictConstructor; use feature qw( signatures postderef postderef_qq ); no warnings qw( experimental::signatures experimental::postderef ); use namespace::autoclean; use mro 'c3'; has '+x' =&gt; ( is =&gt; 'ro', writer =&gt; 'set_x', clearer =&gt; 'clear_x', default =&gt; 0, ); has '+y' =&gt; ( is =&gt; 'ro', writer =&gt; 'set_y', clearer =&gt; 'clear_y', default =&gt; 0, ); sub invert ($self) { my ( $x, $y ) = ( $self-&gt;x, $self-&gt;y ); $self-&gt;set_x($y); $self-&gt;set_y($x); } __PACKAGE__-&gt;meta-&gt;make_immutable; 1;  Here it is in MooseX::Extended :  package My::Point::Mutable; use MooseX::Extended; extends 'My::Point'; param [ '+x', '+y' ] =&gt; ( writer =&gt; 1, clearer =&gt; 1, default =&gt; 0 ); sub invert ($self) { my ( $x, $y ) = ( $self-&gt;x, $self-&gt;y ); $self-&gt;set_x($y); $self-&gt;set_y($x); }  Again, it lets me focus on writing code that works , rather than all the scaffolding that’s usually needed.  Note: we also provide MooseX::Extended::Role  which behaves similarly.  param and field  You can still use the has function to declare your Moose attributes, but I recommend using param and field instead. Why? We’ve regularly face the following problem:  package Some::Class; use Moose; has name =&gt; (...); has uuid =&gt; (...); has id =&gt; (...); has backlog =&gt; (...); has auth =&gt; (...); has username =&gt; (...); has password =&gt; (...); has cache =&gt; (...); has this =&gt; (...); has that =&gt; (...);  Which of those should be passed to the constructor and which should not? Just because you can pass something to the constructor doesn’t mean you should . Unfortunately, Moose defaults to “opt-out” rather than “opt-in” for constructor arguments. This makes it really easy to build objects, but means that you can pass unexpected things to the constructor and it won’t always work the way you want it to.  There’s an arcane init_arg =&gt; undef pair to pass to each to say “this cannot be set via the constructor,” but many developers are either unaware of this is simply forget about it. MooseX::Extended solves with by separating has into param (allowed in the constructor, but you can also use default or builder ) and field , which is forbidden in the constructor.  We can rewrite the above as this:  package Some::Class; use MooseX::Extended; param name =&gt; (...); param backlog =&gt; (...); param auth =&gt; (...); param username =&gt; (...); param password =&gt; (...); field cache =&gt; (...); field this =&gt; (...); field that =&gt; (...); field uuid =&gt; (...); field id =&gt; (...);  And now you can instantly see what is and is not intended to be allowed in the constructor.  There’s a lot more I can say about that, but you can read the manual  for more information.  Attribute Shortcuts  When using field or param , we have some attribute shortcuts:  param name =&gt; ( isa =&gt; NonEmptyStr, writer =&gt; 1, # set_name reader =&gt; 1, # get_name predicate =&gt; 1, # has_name clearer =&gt; 1, # clear_name builder =&gt; 1, # _build_name ); sub _build_name ($self) { ... }  These can also be used when you pass an array reference to the function:  package Point { use MooseX::Extended types =&gt; ['Int']; param [ 'x', 'y' ] =&gt; ( isa =&gt; Int, clearer =&gt; 1, # clear_x and clear_y available default =&gt; 0, ); }  Note that these are shortcuts and they make attributes easier to write and more consistent. However, you can still use full names:  field authz_delegate =&gt; ( builder =&gt; '_build_my_darned_authz_delegate', );  Cloning  MooseX::Extended offers optional, EXPERIMENTAL support for attribute cloning, but differently from how we see it typically done. You can just pass the clone =&gt; 1 argument to your attribute and it will be clone with Storable ‘s dclone function every time you read or write that attribute, it will be cloned if it’s a reference, ensuring that your object is effectively immutable.  If you prefer, you can also pass a code reference or the name of a method you will use to clone the object. Each will receive three arguments: $self, $attribute_name, $value_to_clone . Here’s a full example, taken from our test suite.  package My::Event; use MooseX::Extended types =&gt; [qw(NonEmptyStr HashRef InstanceOf)]; param name =&gt; ( isa =&gt; NonEmptyStr ); param payload =&gt; ( isa =&gt; HashRef, clone =&gt; 1, # uses Storable::dclone writer =&gt; 1, ); param start_date =&gt; ( isa =&gt; InstanceOf ['DateTime'], clone =&gt; sub ( $self, $name, $value ) { return $value-&gt;clone; }, ); param end_date =&gt; ( isa =&gt; InstanceOf ['DateTime'], clone =&gt; '_clone_end_date', ); sub _clone_end_date ( $self, $name, $value ) { return $value-&gt;clone; } sub BUILD ( $self, @ ) { if ( $self-&gt;end_date &lt; $self-&gt;start_date ) { croak(&quot;End date must not be before start date&quot;); } }  Customization  Not everyone likes MooseX::StrictConstructor . It’s useful to prevent this error:  my $name = My::Name-&gt;new( naem =&gt; 'Ovid' );  In bog-standard Moose, the misspelled attribute will be discarded, leading to bugs that can be hard to track down. However, some code (such as DBIx::Class  ), breaks with a strict constructor. You can skip that:  use MooseX::Extended excludes =&gt; [qw\/StrictConstructor\/];  Both MooseX::Extended and MooseX::Extended::Role have a variety of features you can exclude if they don’t work for you. Again, see the documentation.  Note that at the present time, this customization is per class and cannot be inherited. This may change in a future release.  Conclusion  There’s a lot more packed into MooseX::Extended , but this covers the highlights. I think MooseX::Extended will make many Perl programmer’s life easier if they use Moose, but want a better set of defaults. It’s still in ALPHA, but it' close to a feature-complete BETA release. I think it’s relatively solid, but no guarantees. The github repo  has Linux and Windows CI workflows set up to catch errors quickly and the test suite is coming along nicely.  If you’re interested in starting any new projects using Moose, please give MooseX::Extended a try and let me know what you think.  ","title":"Introducing MooseX::Extended for Perl","url":"\/articles\/introducing-moosexextended-for-perl.html"},{"body":"    Introduction  Programming in the Wrong Language  I Failed at Corinna  What I Was Trying to Do  How I Fixed It  Types (or the lack thereof)  Exceptions  Conclusion     Introduction  This post is mainly my findings on finally porting some real code to Corinna, but I confess I was concerned when I first started.  I recently tried to convert a bless object hierarchy to Corinna  and I failed, badly. The code worked, but the design was a mess. As the lead designer of Corinna, after years of effort, I found that Corinna was flawed. After hours of trying to write and rewrite the explanation of the flaw, I found the flaw: me.  Corinna is not flawed. I was thinking in terms of blessed hashes, not Corinna. When I realized what was going on, my thoughts shifted. I started to see the old code was riddled with flaws. Admittedly, the code I was porting was something I designed (*cough*) two decades ago, but it was still humbling.  Programming in the Wrong Language  It used to be that plenty of C developers flocked to the Perl language because it’s so quick-n-easy to whip up a prototype and test out an idea. Of course, experienced Perl programmers would point out that the C programmers were writing Perl as if it was C, not Perl. They’d look at this:  for ( my $i = 0; $i &lt;= @array; $i++ ) { my $element = $array[$i]; ... }  By the way, did you spot the bug?  And they’d say, “just use the elements directly”:  for my $element (@array) { ... }  That easier to read, more likely to be correct, and it’s faster.  In the 90s, I didn’t have that problem when I started with Perl. At the time, it had been over a decade since I had done any serious C hacking, so I didn’t write my Perl like C. I wrote it like COBOL. Seriously! I’d declare all of my variables at the top of the program, as if I were setting up my WORKING-STORAGE SECTION  . Then I’d set the appropriate variables and call the procedure subroutine I needed, though I at least returned data instead of setting a global.  Needless to say, my early days in writing Perl were pretty awful, but it was still a hell of a lot easier than COBOL.  Today, many Perl developers are excited about the Corinna project, for which I have written a short tutorial.  Just as one should stop thinking in C or COBOL when writing Perl, one should stop thinking in terms of using bless  when writing Corinna. If that seems like it’s not too hard, I can assure you, many will stumble as I did.  I Failed at Corinna  My HTML::TokeParser::Simple  module is modestly popular. There are over 30 distributions which depend on it  and I frequently see it used in clients I assist, and given that it’s been around for over two decades with no bugs reported, I’m pretty happy with that module. So when Paul Evans created a PR for a subset of Corinna  , I thought it was time to port something. Instead of mocked-up code in the RFC  or my experiments with Object::Pad  , I was going to write real Corinna code.  I had been following along and giving Paul some feedback on development work. I found a couple of bugs (not many, which is impressive), but now it was time to really push things. Hell, as the lead designer of the Corinna project, based on my original 2019 (mis)design  and that was based on research and work I had done prior to that , I’m pretty comfortable with saying that if anyone can port something to Corinna, I am that guy.     And that’s when I discovered that I’m Prax, not Amos. I am not that guy. (If you haven’t watched or read The Expanse, you’re missing out. Trust me.)  What I Was Trying to Do  The point of this module is that HTML::TokeParser  could parse HTML into a stream of tokens which look like this:  [&quot;S&quot;, $tag, \\%attr, \\@attrseq, $text] [&quot;E&quot;, $tag, $text] [&quot;T&quot;, $text, $is_data] [&quot;PI&quot;, $token0, $text] [&quot;C&quot;, $text] [&quot;D&quot;, $text]  The code I wrote using that was constantly breaking, so I blessed those and put an OO wrapper on them so that $token-&gt;as_is (renamed to $token-&gt;to_string in my new code) always returned what it was supposed to return, instead of using $token-&gt;[4] (start tag), $token-&gt;[2] (end tag), $token-&gt;[1] (text), an so on. You can’t even use $token-&gt;[-1] to read the last item thanks to the T token (text) ironically being the token which had the plaintext in an unpredictable position.  That’s much easier than using HTML::TokeParser directly. I did this by calling bless with the array references and blessing them into appropriate classes. This meant the array reference was still available and HTML::TokeParser::Simple was a drop-in replacement for the original module. You could switch from HTML::TokeParser to HTML::TokeParser::Simple with no other changes in your code. You then gradually converted array reference lookups to method calls. I was doing a lot of web scraping in the pre-API days of the Web and this saved me much grief.  So when I started designing HTML::TokeParser::Corinna  , I hit my first snag.  Since Corinna is designed to be encapsulated, you can’t call $token-&gt;[1] . No “reaching inside” the object is allowed. But that’s fine! Since HTML::TokeParser::Corinna is a new module, I can create any interface I want. That’s when I hit my next problem.  For each of the array reference types listed above, I have a corresponding class:   HTML::TokeParser::Corinna::Token::Tag::Start  HTML::TokeParser::Corinna::Token::Tag::End  HTML::TokeParser::Corinna::Token::Text  HTML::TokeParser::Corinna::Token::Comment  HTML::TokeParser::Corinna::Token::Declaration  HTML::TokeParser::Corinna::Token::ProcessInstruction   There are some common behaviors there and since we don’t yet have roles for Corinna, I used abstract base classes. (We’ll shorten the prefix to the namespace to make it easier to read):   HTC::Token  HTC::Token::Tag :isa(HTC::Token)   I can instantiate a corresponding class like this, with all constructors having the same interface:  my $end_tag = HTC::Token::Tag::End-&gt;new( token =&gt; $token );  Since HTC::Token is the base class for everything, I have this:  class HTC::Token { field $token :param; method _get_token () {$token} ... }  It also has the methods common to all token classes.  Subclasses look like this:  class HTC::Token::Tag :isa(HTC::Token) { ... } class HTC::Token::Tag::Start :isa(HTC::Token::Tag) { ... }  Even ignoring the fact that my objects were mutable, my code is flawed. The “Tag” classes need to be able to access the $token from the parent class. I have no way to do that, so I have a _get_token method. Untrusted code can call $token-&gt;_get_token and change the array reference in unexpected ways. That kills one of the major points of Corinna, but I’ve no easy way of sharing that data otherwise.  Realizing I could not fix this was my crushing blow, leading me to naïvely believe Corinna was flawed. What follows is how I worked through the issue, but it took longer for me to have clarity than what is shown here.  How I Fixed It  One way of handling this is the following:  class HTC::Token { field $token :param; method _get_token () {clone($token)} ... }  But that still leaves _get_token() callable outside the class and it’s now part of the interface. It becomes an implementation detail I don’t have the freedom to change (classes should be open for extension, not modification). It’s part of the class contract and should not be violated.  Corinna doesn’t have a clean way of handling this case, but it’s not a flaw. It’s a limitation and one we can easily fix. Adding a :trusted attribute  to methods would make this much easier, but that’s still an open discussion.   A trusted method, whether provided by an abstract base class or a role, should propagate to the first non-abstract subclass and become a private method in that class. If it’s defined directly in a concrete (non-abstract) class, then the first concrete class which inherits it gains it as a private method.   This isn’t quite how trusted methods work in other languages, but that’s OK. Perl is not like other languages and we have to adapt.  Lacking trusted methods, I cut-n-pasted the field $token :param; line into each of my concrete classes. That solved the problem, but if they each took multiple parameters, having to sync them across multiple classes would be fragile. A single role (or base class) providing those as :trusted would make this issue go away.  So, bullet dodged. Corinna isn’t irrevocably broken, but it did give me a bit of a scare at first. However, it also pleased me. “No plan survives first contact with the enemy,” but I confess I had a small concern. Despite years of research and design, maybe we had missed something critical. Finding only a tiny limitation has been a relief (though whether this holds remains to be seen).  Types (or the lack thereof)  This next part isn’t about a limitation of Corinna, but of my not understanding object-oriented design when I first wrote HTML::TokeParser::Simple . This is related to type theory.  A type system is nothing more than a way of evaluating expressions to ensure they do not produce unwanted behavior. Perl’s primary type system is based on data structures, not data types .   For example, you can’t access an array element the way you access a hash element (though Perl being Perl, there are ways you can change that, too). But what is two plus two? We know that the answer is four, yet the computer often needs help. Let’s look at the following:  my @array = qw(10 11 12); my $var = \\@array; say 2 + 2; # Int + Int say 2 + &quot;2&quot;; # Int + String say &quot;2 pears&quot; + &quot;2 weeks&quot;; # Str + Str say 2 + @array; # Int + Int! say 2 + $var; # Int + Ref  That prints something like:  4 4 Argument &quot;2 weeks&quot; isn't numeric in addition (+) at two.pl line 8. Argument &quot;2 pears&quot; isn't numeric in addition (+) at two.pl line 8. 4 5 5201037554  For many languages, only 2 + 2 would be valid in the above. Perl is heavily optimized for text manipulation, so if you’re reading in a bunch of data from a text file, you can often treat numeric strings as numbers. Thus, 2 + \"2\" is 4 . The ASCII value of \"2\" is 50, but Perl understands what you mean and casts the string as an integer instead of calculating 2 + 50 .  The \"2 pears\" + \"2 weeks\" is clearly nonsense, but at least you get a warning.  2 + @array surprises many people new to Perl, but it’s evaluating @array in scalar context. Since it has three elements, this reduces to 2 + 3 , printing 5 . I know several programmers who write this as 2 + scalar @array to be explicit about the intent.  But what’s with that 5201037554 in the output? Your number will vary if you run this code, but what’s happening is that $var , in the expression 2 + $var , evaluates to the address of the reference \\@array . You don’t even get a warning. This is useless (no pointer math in Perl) and yes, I’ve been bitten by this in production code.  For many languages this expression would prevent your program from compiling, but Perl is Perl. For the poor maintenance programmer seeing my $result = $var1 + $var2; buried deep in the code, it may not be immediately obvious there’s an issue.  So this gets us back to a fundamental question: what is a type? A type is nothing more than:   A name for the type  A set of allowed values for that type  A set of operations allowed to be called on that type   If we think of an integer as an object and addition as a method, let’s play with some pseudocode and pretend we have multimethods and a way of declaring data types.  class Int { field $int :isa(Int); multimethod add ($value :isa(Int)) { return $int + $value; } multimethod add ($value :isa(Str) :coerce(Int)) { return $int + $value; } } my $int = Int-&gt;new( int =&gt; 2 ); say $int-&gt;add(3); # 5 say $int-&gt;add(&quot;4&quot;); # 6 # runtime error because we can't coerce say $int-&gt;add(&quot;4 apples&quot;); # problematic because arrays flatten to lists and # an array with one element will work here, but # zero or two or more elements are fatal say $int-&gt;add(@array); # fatal because there is no multimethod dispatch target say $int-&gt;add(\\@array);  In the above, we simply don’t provide methods for behaviors we don’t want. Yes, the developer may very well have to check that they’re not passing in bad data, but this is not a bad thing. At their core, objects are experts about a problem domain and you need to take care to get them right.  This also fits with the principle that we want to minimize our interfaces as much as much as possible. The more methods you expose, the more methods you have to maintain. If you later need to change those methods, you may break existing code. So let’s look at my abstract HTC::Token base class, a more-or-less straight port of the original code:  class HTML::TokeParser::Corinna::Token { field $token : param; method to_string { $token-&gt;[1] } method _get_token {$token} method is_tag {false} method is_start_tag {false} method is_end_tag {false} method is_text {false} method is_comment {false} method is_declaration {false} method is_pi {false} method is_process_instruction {false} method rewrite_tag { } method delete_attr { } method set_attr { } method tag { } method attr (@) { {} } method attrseq { [] } method token0 { } }  That ensures that every class has a stub of every method available to it, so you won’t get a “method not found” error. But what if you have a token representing text in your HTML? Why on earth would you want to call $token-&gt;rewrite_tag if it’s not a tag? It’s like the above example of adding an integer to a reference: you can do it, but it’s not helpful.  What is helpful is knowing what kind of token you have. So my base class is now:  class HTML::TokeParser::Corinna::Token { method is_tag {false} method is_start_tag {false} method is_end_tag {false} method is_text {false} method is_comment {false} method is_declaration {false} method is_pi {false} method is_process_instruction {false} }  This is cleaner and easier to maintain. In fact, I could delete this class, but those predicate methods are much easier to use.  if ( $token isa 'HTC::Token::Tag::Start' ) { ... } # versus if ( $token-&gt;is_start_tag ) { ... }  I’ve also taken the trouble to make all tokens immutable. We generally want immutable objects , but in reality, sometimes it’s cumbersome. If you want to replace the class newz with news everywhere, here’s what you do:  my $parser = HTC-&gt;new( file =&gt; $file ); while ( my $token = $parser-&gt;next ) { if ( $token-&gt;is_start_tag &amp;&amp; $token-&gt;attr('class') eq 'newz' ) { $token = $token-&gt;set_attrs( class =&gt; 'news' ); } print $token-&gt;to_string; }  The mutators such as set_attrs now return a new instance instead of mutating the token directly. That makes it safer because you don’t worry about unrelated code mutating your data. For example, if you call $object-&gt;munge(3) , you never worry that the value of 3 has suddenly changed in your code. However, $object-&gt;munge($other_object) offers no such guarantee.  In the code snippet above, however, always having to remember to assign the return value feels, well, clumsy. In fact, if you call set_attrs in void context (i.e., you don’t assign the return value to anything), the code will throw a HTML::TokeParser::Corinna::Exception::VoidContext exception (yes, it now has true exceptions, but they’re part of this module, not part of Corinna).  So my interfaces are smaller and we no longer provide useless, potentially confusing methods. I think that’s a win.  Exceptions  As a final note, I was proud of an odd little trick. I wanted to use exceptions as much as possible. They fix a very common bug in production code. If someone calls die or croak , you often see code like this:  try { some_code(); } catch ($e) { if ( $e =~ \/connection gone away\/ ) { # retry connection or rethrow exception } die $e; };  Now if some maintenance programmer renames the error message to connection interrupted , all code dependent on the previous error message breaks. But if they throw an exception in an Exception::Connection::Lost class, the code can check the class of the exception and the developers are free to change the actual error message any way they like.  So here’s my exception base class:  class HTML::TokeParser::Corinna::Exception { use overload '&quot;&quot;' =&gt; 'to_string', fallback =&gt; 1; use Devel::StackTrace; field $message :param = undef; field $trace = Devel::StackTrace-&gt;new-&gt;as_string; method error () {&quot;An unexpected error occurred&quot;} method message () {$message} method stack_trace () {$trace} method to_string { # error() can be overridden my $error = $self-&gt;error; # but $message is universal if ($message) { $error .= &quot;\\n$message&quot;; } return &quot;Error: $error\\n\\nStack Trace:\\n$trace&quot;; } }  Because stringification is overloaded, I can print the exception or check it with a regex. Because it’s an object, you can check the class of the exception to decide what to do next.  I used the above to provide a MethodNotFound exception:  class HTC::Exception::MethodNotFound :isa(HTC::Exception) { field $method :param; field $class :param; method error () { &quot;No such method '$method' for class '$class'&quot; } method method () {$method} method class () {$class} }  And in my base class, I have this:  method AUTOLOAD { our $AUTOLOAD; my ( $class, $method ) = ( $AUTOLOAD =~ \/^(.*)::(.*)$\/ ); return if $method eq 'DESTROY'; throw( 'MethodNotFound', method =&gt; $method, class =&gt; $class, ); }  And now, $token-&gt;no_such_method throws an exception instead of causing you to die inside.  Conclusion  The earlier description of the hours of writing and rewriting to explain the flaw encompass much more than what I’ve discussed, but I wanted to keep this short. Of course, I threw in a few other things I noticed along the way.  The encapsulation violation seemed to break the main strength of Corinna, but spending a few hours porting a class hierarchy quickly exposed the design limitation and a solution presented itself. Perhaps the Corinna design team or someone else might offer a more elegant solution than what I presented. I’m OK with that.  So far, the Corinna code is simpler, easier to read, provides strong encapsulation, and was generally a pleasure to write. I’m looking forward to the day it’s production ready. I expect there will be teething pain for others, since thinking in terms of blessed hashes is ingrained in Perl culture, but if we keep living in the past, Perl will become a thing of the past.  The first PR for use feature 'class' is here  and I’m sure any and all feedback would be appreciated.  ","title":"Converting Object-Oriented Code to Corinna","url":"\/articles\/converting-object-oriented-code-to-corinna.html"},{"body":"    Quickstart  The Overview  git-flow  Our git Workflow  Assumptions  Checking Out a Branch  Caveats  Refreshing Your Branch  Merging into master  Determining Your Primary Branch  Merge Conflicts  Update        We say what we do and do what we say.    Note : For historical reasons, we have left the name master in a number of places in this document. Many projects prefer that this name be main or something different.   One of the things we constantly strive to do at All Around the World  is apply the Pareto Principle  : 80% of your results stem from 20% of your actions. When we go in to solve problems for clients; we don’t have the luxury of sitting around for days to debate the perfect way of doing things before we start working. Instead, we strive to find the 20% of actions which give us 80% of the results and uncover the rest as we go. Today, I’ll talk about our git workflow  and how three simple commands make our team collaboration much easier. Email us to discuss how we can build a remote software team for you. A team that delivers, not debates.  Quickstart  Using our git workflow couldn’t be simpler:  (master) $ git hub 123 # create new branch (new-feature-123) $ # hack, hack, hack (new-feature-123) $ git refresh # pull in latest master changes (new-feature-123) $ # hack, hack, hack (new-feature-123) $ git done (master) $ # done  And that’s it. Three simple commands to remember and git gets out of your way .  The Overview  The git command is extremely complicated. Invariably in a team environment someone gets something “wrong” with git and the local git guru comes by and does mysterious things with git fsck , git reflog , and other commands that developers have never heard of. And while it’s useful to have a git guru on hand, we find it’s even easier to have a single, standard way of using git so that all developers are doing things the same way. Even developers new to git find our workflow very easy.  It’s designed largely to avoid this:    A typical git history.   For us, the complicated history was the showstopper. We want a clean, linear history and an “easy-to-use” workflow. It’s based on the following logic:   The master branch is the source of truth  All new development happens in a new branch off of master  When you’re done, merge your code back into master and delete your branches   In other words, it’s very clean, very simple, and focuses on the core development needs. There’s nothing surprising about this workflow. How do releases work? That depends on your project and its needs. How do hotfixes work? That depends on your project and its needs. How do code reviews work? That depends on your project and its needs. Like the WildAgile project management system , our workflow is designed to be the core of what you do day in and day out. The bits you need to customize are up to you. We’re not dogmatic.  git-flow  As an aside, one popular workflow is git-flow  . Yet git-flow has its detractors  and it’s easy to find many more  . We found the convoluted history and strict requirements that things be “just so” to not be a good fit for our workflow. It might be the perfect fit for yours.  Our git Workflow  The git workflow tools  are open source and they contain a simplified subset of the git tools used by All Around the World  for our software development. It makes it dead easy for teams using git (and in our case, github) to work together.  There are only three new commands to remember:   git hub $issue  Create new branch based on a github ticket  git refresh  Rebase your work on top of current master  git done  Merge your branch back into master   Note: only the bin\/git-hub command assumes you’re using github. The other commands work fine without it.  Assumptions  The master branch is never worked on directly. Instead, new branches are created (usually for an individual github ticket), we hack, we regularly pull new changes into that branch, and after a pull request is created and the approved, we merge the code back into master .  The examples below assume the files in the bin\/ directory are in your path. If they are not in your path, you have to type the commands explicitly:  bin\/git-hub 5738 bin\/git-refresh bin\/git-done  Checking Out a Branch  We assume that branches are usually per ticket. You can manually create a branch, but we tend not to. Instead, if you are going to work on github issue 5738, with the title “Lower reputation for failed crimes”, you run the following command:  git hub 5738  And you’re in a shiny new branch named lower-reputation-for-failed-crimes-5738 .  If that branch already exists, it’s checked out.  If you’re not using github, it’s trivial to do this by hand:  git checkout -b name-of-your-branch  We just like the convenience of always having standard naming conventions.  Caveats   The new branch that is created is based on the branch you’re on, so you probably want to run this from master  Branch detection is based on a heuristic, using the ticket number. If you have more than one branch with that ticket number, you will get a warning and the code will exit.  You’ll need a config file for this. See perldoc bin\/git-hub for this.  Assumes Perl 5 version 20   You will also need to install the following modules from the CPAN  if you’ve not already done so.   autodie  Config::General  JSON  Text::Unidecode  Pithub   If you’re not familiar with installing CPAN modules, check out cpanminus  , or you can use the cpan command line tool that is standard with Perl.  Refreshing Your Branch  While working on your branch, you want to regularly pull in the latest master to keep your code up-to-date. Working on a change for a week with a fast-moving codebase can cause serious headaches when it’s time to merge. Thus, you should regularly run the following in your branch (I try to run it at least once per day):  git refresh  Regardless of the branch you are on, this code:   Stashes changes (if any)  Checks out master  Does a fast-forward merge  Checks out your branch (if branch is not master)  Rebases onto master (if branch is not master)  Pops changes from stash, if any   In other words, it cleanly rebases your code on top of master, even if you have uncommitted changes.  Note: for everyone who keeps asking “why not git pull --rebase --autostash ?”  The simple reason is backwards-compatibility. Sometimes developers have older versions of git installed and while that’s usually perfectly compatible with newer versions, we don’t force them to upgrade. Also, internally git-refresh does things like git fetch -p to remove references to remote branches which no longer exist. This regular “house cleaning” helps to keep git more performant. See the Pruning  documentation for git-fetch for more information.  Note: if this command aborts (such as during a merge conflict), the stashed changes will remain stashed. Run git stash to see those changes and git stash pop to return them to your codebase when you’ve finished resolving the merge conflicts.  Merging into master  So you’ve finished your work and are ready to merge your branch back into master. Here’s one way to do that very cleanly and safely:  git checkout master git fetch --prune git merge --ff-only origin\/master git rebase master my-awesome-feature-1234 git push --force-with-lease origin my-awesome-feature-1234 git checkout master git merge --no-ff my-awesome-feature-1234 git push origin master  You’ll remember that, right? And you’ll never mess it up?  For us, we simply run:  git done  And that will cleanly update master (or whatever the source branch is) and rebase your branch on top of it, and push that change to your origin.  With that, you get a clean git history like this:  | * 44eba1b094 - ... | * 217350810f - ... |\/ * c84e694e59 - Merge branch 'no-add-message-from-context-object-8615' PR#8622 (6 days ago) &lt;some author&gt; |\\ | * 9d73143f75 - ... | * 983a1a5020 - ... | * e799ecc8e3 - ... | * aa9c981c2e - ... | * 4651830fd6 - ... |\/ * 010e94b446 - Merge branch 'fix-test-combat-module-usage' PR#8623 (7 days ago) &lt;some author&gt; |\\ | * 46d8917af7 - ... |\/ * 4acfdd8309 - Merge branch 'economy-action-use-args-hashref' PR#8617 (10 days ago) &lt;some author&gt; |\\ | * a1f863f908 - ... | * b3dc3efb2a - ... | * ab77373fca - ... | * b5491e4ae9 - ...  And when you’re done, it will also print out the instructions on how you can delete your local and remote copies of the branch, to help you be nice and not leave branches cluttering up the repository.  Determining Your Primary Branch  Internally the code attempts to detect the name of your primary branch using this:  basename $( git symbolic-ref refs\/remotes\/origin\/HEAD )  That will usually work. For all commands, if it does not work, it will print out the following message:  Could not determine target branch via '&quot;basename $( git symbolic-ref refs\/remotes\/origin\/HEAD )&quot;' You scan set your target branch with: git symbolic-ref refs\/remotes\/origin\/HEAD refs\/remotes\/origin\/\\$branch_name Where $branch_name is the name of the primary branch you develop from ('main, 'master', etc.)  Merge Conflicts  With all of this rebasing, you’re sometimes going to hit the dreaded ...  Auto-merging lib\/Veure\/Moose.pm CONFLICT (content): Merge conflict in lib\/Veure\/Moose.pm Failed to merge in the changes. Patch failed at 0001 Issue 144 Make read-only the default for attributes  And then you’re very unhappy. Worse, if you do this after a week of isolated hacking, you might get a huge amount of merge conflicts to work through. However, by running git refresh at least once a day, if not more often, those conflicts are minimized. And when you do get them? I have the following in my .bash_aliases file:  alias damnit='vim $(git grep -l \"&lt;&lt;&lt;&lt; HEAD\")'  Then, when I get a conflict, I just type damnit and am automatically editing the files which caused the conflict. It saves me a huge amount of time.  Update  We have renamed git-merge-with-master to git-done . The former still works.  Check out the tools at our github repo  .  ","title":"Easy Git Workflow","url":"\/articles\/easy-git-workflow.html"},{"body":"  When I was working with the BBC, the age-old debate had arisen again: should we use “cuddled elses” or not?   For those not familiar with the debate, it’s asking if we should do this:  if (someCondition) { System.out.println(&quot;We all gonna die&quot;); } else { System.out.println(&quot;We not gonna die&quot;); }  Or this:  if (someCondition) { System.out.println(&quot;We all gonna die&quot;); } else { System.out.println(&quot;We not gonna die&quot;); }  For those of you who are not software developers, the differences might seem pedantic. For those of you who are software developers, the difference is pedantic. Unfortunately, it caused a furious debate (as it often does) at the BBC and, as often happens with these debates, work isn’t done. Sometimes tempers flare. And quite often a decision isn’t made.  And that’s actually bad because one developer might “cuddle” that else and another developer working on this code might “uncuddle” it. And they’re not being malicious or petty: “correctly” formatting code is simply something an experienced developer does as part of their job. And that means more commits to your version control system and more work digging through those commits if you’re trying to figure out why something was written in a particular way.  Or worse, you have a developer who decides to write code to format the entire codebase in their personal style and running git annotate $filename shows that developer on all the lines. So you contact this “expert” about the $filename and they say they’ve never heard of it. I’ve run across this mess more than once.  Getting back to the BBC ...  We had agreed we’d use Perl::Tidy  to automatically format our code, but at that point, we hit another obstacle because everyone argued about the brace style (if you don’t know what that means, trust me, you don’t care) and no matter how hard we tried, Perl::Tidy couldn’t get the curly braces formatted the way we wanted them to be formatted.  My argument was “who cares? We’ll get used to the new styles. Let’s get back to work.” But another developer, passionate about his brace placement, argued that he could write a PPI  hook to fix this. Sigh.  I quietly bet another developer £5 that this approach would not work. I’m sad to say I won that bet. Worse, when the “solution” was ready, it kept hitting edge cases in our code and breaking.  And if you really want to see what a ridiculous timesink this can be, you can choose many different indentation styles such as K&amp;R (and its many variants), Allman, Pico, Ratliff, GNU, ad nauseum   What a waste of time!  The icing on the cake? Though we know having regular formatting of code to show the logical structure of the program makes it easier to follow the code (proof: read minimized Javascript versus unminimized), there’s no strong evidence (that I’ve ever found) a particular type of formatting is better than another. If you disagree, please provide non-anecdotal evidence.  In fact, in the excellent book Code Complete: A Practical Handbook of Software Construction, Second Edition  , by Steve McConnell, he dives into this at the beginning of chapter 31. After some discussion of the topic, including referencing some studies on said topic, he concludes (emphasis mine):   Expert programmers often cling to their own styles tenaciously, even when they’re vastly different from other styles used by other expert programmers. The bottom line is that the details of a specific method of structuring a program are much less important than the fact that the program is structured consistently .   Unsurprisingly, these arguments about brace style, cuddled elses, tabs versus spaces,   and so on, are almost meaningless. To be fair, there is marginal value to be had in suggesting things such as “shorter variable names for smaller scopes” and “indent to show logical structure”, but being too pedantic about it pisses people off for no good reason.  So how do we fix this?  First, pick a reasonable style you can automate , swallow your pride about the details, and choose your automation tools. Second, consider that—unless you’re pair programming—noone really cares much about the font you’re using, do they? Not really. You’ve set your editor to choose your own font and that’s not written back to your source code, so set your editor to choose your own style and make sure it’s not written back to the source code.  So let’s say you prefer spaces to tabs, with a four-space indent, and you use vim to edit your code. When you open a file, or create a new one, using the BufRead,BufNewFile commands is the first thing you need, or you can create a mapping to apply your style. For example, in your filetype mappings you can add the following:  noremap &lt;silent&gt; &lt;leader&gt;style :call s:ApplyMyStyle()&lt;cr&gt; function! s:ApplyMyStyle() &quot; set up my editor to do the right thing setfiletype perl set expandtab set textwidth=78 set tabstop=4 set shiftwidth=4 set expandtab retab &quot; apply my tab\/space choices &quot; reformat the file execute(&quot;:%!perltidy -q --profile=myperltidyrc&quot;) endfunction  Now, assuming your leader is the backslash, you can type \\style to apply your coding style and never, ever have to pay attention to the silly choices other developers make.  Do I recommend this technique? Absolutely not. You’re going to commit in your own style one day. However, there are git hooks you can write to apply styles when you commit  . It’s much safer. A better choice, still, is to just be an adult and admit your “one true style” is a personal preference, not a mandate handed down from heaven. Learn to use the house style in preference to your own and avoid all of the petty arguments in the first place. You won’t look like a better person for it, but you’ll look like a worse one if you tell the other programmers they’re fools for choosing a style different from your own.  ","title":"Automate Your Software Standards","url":"\/articles\/automated-software-standards.html"},{"body":"  The Problem  Test databases are very easy to get wrong. Very easy. Decades ago when I first learned testing, the team shared a test database. If you ran your test at the same time another developer, both of your test suites would fail! However, we were using a database where we had to pay for individual licenses, so we were limited in what we could do.  Later, I worked for a company using MySQL and I created an elaborate system of triggers to track all database changes. This let me “fake” transactions by starting a test run, see what had changed last time, and automatically reverting those changes. It had the advantage that multiple database handles could see each other’s changes (hard to do for many databases if you have separate transactions). It had the disadvantage of everything else: it was fragile and slow.  Later, I started using xUnit frameworks, eventually writing a new one  that’s popular for companies needing a large-scale testing solution. With this, it was easy for each test class to run in a separate transaction, cleaning itself up as it went. Using transactions provides great isolation, leverages what databases are already good at, and let’s you run many classes in parallel.  But it can easily break embedded transaction logic. And you have to guarantee everything shares the same database handle, and you can’t really test the transactions in your code, and, and, and ...  What finally drove me over the edge was writing some code for a client using the Minion job queue  . The queue is solid, but it creates new database connections, thus ensuring that it can’t see anything in your database transactions. I figured out a (hackish) solution, but I was tired of hackish solutions.  While I was researching the solution, Matt Trout was reminding me (again) why the “database transactions for tests” approach was broken. Just spawn off temporary test databases and use those, throwing them away when you’re done.  The Client  So a company wanting to hire me gave me a technical test and there was a task to add a small, simple feature to a Catalyst web application  . It was trivial. They handed me a Vagrant file and after a quick vagrant up and vagrant ssh , I was ready to begin. Then I looked at the test they had for the controller:  use strict; use warnings; use Test::More; use Catalyst::Test 'Client'; ok( request('\/some_path')-&gt;is_success, 'Request should succeed' ); done_testing();  The task involved a POST to a URL. There was no test for the existing feature that I was adding to, but any test I wrote meant I’d be changing the state of the database. Run the code multiple times and I’d leave junk in the database. There were various ways I could approach this, but I decided it was time to build a quick database on the fly, write to that, and then dispose of it after. The code for this was trivial:  package Test::DB; use File::Temp qw(tempfile); use DBI; use parent 'Exporter'; use Client::Policy; BEGIN { if ( exists $INC{'Client\/Model\/ClientDB.pm'} ) { croak(&quot;You must load Test::DB before Client::Model::ClientDB&quot;); } } use Client::Model::ClientDB; our @EXPORT_OK = qw(test_dbh); my $connect_info = Client::Model::ClientDB-&gt;config-&gt;{connect_info}; my $dsn = $connect_info-&gt;{dsn}; my $user = $connect_info-&gt;{user}; my $password = $connect_info-&gt;{password}; # $$ is the process id (PID) my $db_name = sprintf 'test_db_%d_%d', time, $$; my ( $fh, $filename ) = tempfile(); my $dbh = DBI-&gt;connect( $dsn, $user, $password, { RaiseError =&gt; 1, AutoCommit =&gt; 1 } ); $dbh-&gt;do(&quot;CREATE DATABASE $db_name&quot;); system(&quot;mysqldump -u $user --password=$password test_db &gt; $filename&quot;) == 0 or croak(&quot;mysqldump failed: $?&quot;); system(&quot;mysql -u $user --password=$password $db_name &lt; $filename&quot;) == 0 or croak(&quot;importing schema to mysql failed: $?&quot;); # XXX We’re being naughty in this quick hack. We’re writing # this back to the Model so that modules which use this connect # to the correct database. $connect_info-&gt;{dsn} = &quot;dbi:mysql:$db_name&quot;; # This is just a quick hack to get tests working for this code. # A catastrophic failure in the test means this might not get # run and we have a junk test database lying around. # Obviously we want something far more robust END { $dbh-&gt;do(&quot;DROP DATABASE $db_name&quot;) } sub test_dbh () { $dbh } 1;  The above is very naughty in many ways, but the client hinted that how fast I returned the test might be a factor (or maybe they didn’t and I misread the signals). They also made it clear they were looking at how I approached problems, not whether or not the code was perfect. Thus, I thought I was on safe territory. And it meant I could do this in my test:  use strict; use warnings; use Test::More; use lib 't\/lib'; use Test::DB; use Catalyst::Test 'Client'; ok( request('\/some_path')-&gt;is_success, 'Request should succeed' ); # anything I do here is against a temporary test database # and will be discarded when the test finishes done_testing();  The Test::DB code was quick and easy to write and made it trivial for me to safely write tests. I was pleased.  What’s Wrong With Test::DB?  For a junior developer, Test::DB might look awesome. For an experienced developer, it’s terrible. So what would I do to make it closer to production ready?  Here are just a few of the things I would consider.  Stronger Data Validation  First, let’s look at our connection information:  my $connect_info = Client::Model::ClientDB-&gt;config-&gt;{connect_info}; my $dsn = $connect_info-&gt;{dsn}; my $user = $connect_info-&gt;{user}; my $password = $connect_info-&gt;{password};  The above relied on how Catalyst often sets up its DBIx::Class (a Perl ORM) model:  package Client::Model::ClientDB; use strict; use base 'Catalyst::Model::DBIC::Schema'; __PACKAGE__-&gt;config( schema_class =&gt; 'Client::Schema::ClientDB', connect_info =&gt; { dsn =&gt; 'dbi:mysql:test_db', user =&gt; 'root', password =&gt; 'rootpass', } );  Once you load that class, you get a config class method which can tell you how that class is configured. However, there’s no guarantee in the Test::DB side that the data is structured the way that I expect. Thus, I need to validate that data and throw an exception immediately if something has changed.  And how do we create our test database?  $dbh-&gt;do(&quot;CREATE DATABASE $db_name&quot;); system(&quot;mysqldump -u $user --password=$password test_db &gt; $filename&quot;) == 0 or croak(&quot;mysqldump failed: $?&quot;); system(&quot;mysql -u $user --password=$password $db_name &lt; $filename&quot;) == 0 or croak(&quot;importing schema to mysql failed: $?&quot;);  The CREATE DATABASE command is fast, so I’m not worried about that. And the test had a single table with very little data, so this was quick. But for Tau Station  , we have a couple of hundred tables and tons of data. This would be slow. For any reasonably mature system, dumping the database each time would be a bad idea. There are also ways you could easily avoid dumping it multiple times, but that hits the next problem: adding that data to your new test database. That would need to be done for each test and that is not something you can trivially speed up.  For a more robust system, I’d probably create a local database service that would simply build a set number of test databases and have them waiting. The test would request the next test database, the service would regist that the database had been taken, and create a new test database in the background while your test runs. The service would also probably clean up old test databases based on whatever policies you think are appropriate.  No Action At A Distance  This line is terrible:  $connect_info-&gt;{dsn} = &quot;dbi:mysql:$db_name&quot;;  The reason that works is because the config data in Client::Model::ClientDB is global and mutable and $connect_info is merely a reference to that data. Instead, if I have a “database service” that tells the code which database it can use, then Test::DB can call that service, and so can Client::Model::ClientDB . Everything relies on a single source of truth instead of hacking global variables and hoping you don’t mess up.  Don’t Drop The Test Database  If there is one thing which I hate about many testing systems, it’s a watching a test horribly fail, but the database is cleaned up (or dropped) and I can’t see the actual data after the test is done. What I often have to do is fire up the debugger and run the code up to the test failure and grab a database handle and try to inspect the data that way. It’s a mess.  Here, we can fix that by simply dropping this line:  END { $dbh-&gt;do(&quot;DROP DATABASE $db_name&quot;) }  At the beginning and end of every test run, we can diag the test database name and if I need to see if there’s an issue in the database, I can still use it. Our database service would have code to drop the database on:   The next day  The next test run  After exceeding a threshold of databases  ... or whatever else you need   In short, keep the valuable data around for debugging.  Rapid Database Development  The database service solution would also have to tie into your database change management strategy. I heavily use sqitch  to manage database changes and I’ve written a lot of code to support odd edge cases. It wouldn’t be hard to write code to let the database service see if it’s up to date with your local version of sqitch . Whatever database change management strategy you use, it needs to be discoverable to properly automate the database service.  Of course, you think, that’s obvious. Yet you’d be shocked how many times I’ve worked with clients whose database change management strategy involves listing a bunch of SQL files and checking their mtime to see which ones need to be applied to your database. Yikes!  Faster Tests  If this is done well, your tests should also be faster. You won’t have the overhead of transactions beyond what your code already has. Plus, you can avoid issues like this:  sub test_basic_combat_attack_behavior ($test,$) { my $ovid = $test-&gt;load_fixture('character.ovid'); my $winston = $test-&gt;load_fixture('character.winston'); my $station = $test-&gt;load_fixture('station.tau-station'); $test-&gt; move_to($station-&gt;port, $ovid, $winston); ok !$ovid-&gt;attack($winston), 'We should not be able to attack someone on the home station.'; ... }  In the above, we’re loading some fixtures. Sometimes those fixtures are very complicated and loading them takes time. For one client, when I would run $test-&gt;load_all_fixtures('connection'); , that would add an extra couple of seconds to every test which needed to do that.  Instead, pre-built test databases can have the test fixtures already loaded. Further, having a pre-populated database helps your code deal with something closer to a real-world problem instead of dealing with an empty database and not catching corner cases that might cause.  Conclusion  By using a database service which merely hands you a temporary test database, you don’t have to worry about leaving the database a mess, managing transactions in tests, or having nasty hacks in your tests to workaround these issues. Most importantly, you’re not changing the behavior of your code. You just use the database like normal. It might be a little bit more work up front to create that database, but it’s worth the effort.  I really do want to get around to creating a proper database tool like this some day. Today is not that day. But I was delighted how even my quick hack, written in just a couple of minutes, made it so much easier to test my code. I should have done this ages ago.  ","title":"Managing a Test Database","url":"\/articles\/managing-a-test-database.html"},{"body":"   Introduction  Common Test Suite Problems  Tests often emit warnings  Tests often fail  There is little evidence of organization  Much of the testing code is duplicated  Testing fixtures are not used (or poorly used)  Code coverage is poorly understood  They take far too long to run  Recommendations  Aggressively search for and remove duplicated tests.  Use code coverage aggressively  Look for code with \"global\" effects  Inline \"hot\" functions.  Recompile your Perl  Preload modules  Parallel tests  Distributed tests  Devel::CoverX::Covered  Test::Class::Moose     This document is about testing applications&mdash;it's not about how to write tests. Application test suites require a different, more disciplined approach than library test suites. I describe common misfeatures experienced in large application test suites and follow with recommendations on best practices. Much of what I describe below is generic and applies to test suites written in any programming language, despite many examples being written in Perl.  Introduction  I often speak with developers who take a new job and they describe a Web site built out of a bunch of separate scripts scattered randomly through directories, lots of duplicated code, poor use of modules, with embedded SQL and printing HTTP headers and HTML directly. The developers shake their head in despair, but grudgingly admit an upside: job security. New features are time-consuming to add, changes are difficult to implement and may have wide-ranging side-effects, and reorganizing the codebase to have a proper separation of concerns, to make it cheaper and safer to hack on, will take lots and lots of time.  A bunch of randomly scattered scripts, no separation of concerns, lots of duplicated code, poor use of modules, SQL embedded directly in them? Does this sound familiar? It's your standard test suite. We're horrified by this in the code, but don't bat an eyelash at the test suite.  Part of this is because much, if not most, of the testing examples we find focus on testing distributions, not applications. If you were to look at the tests for my module DBIx::Class::EasyFixture , you'd see the following tests:  00-load.t basic.t definitions.t groups.t load_multiple_fixtures.t many_to_many.t no_transactions.t one_to_many.t one_to_one.t  These tests were added one by one, as I added new features to DBIx::Class::EasyFixture and each *.t file represents (more or less) a different feature.  For a small distribution, this isn't too bad because it's very easy to keep it all in your head. With only nine files, it's trivial to glance at them, or grep them, to figure out where the relevant tests are. Applications, however, are a different story. This is the number of files from one of my customer's test suites:  $ find t -type f | wc -l 288  That's actually fairly small. One codebase I worked on had close to a million lines of code with thousands of test scripts. You couldn't hold the codebase in your head, you're couldn't glance at the tests to figure out what went where, nor was grepping necessarily going to tell you as tests for particular sections of code were often scattered around multiple test scripts. And, of course, I regularly hear the lament I've heard at many shops with larger codebases: where are the tests for feature X ? Instead of just sitting down and writing code, the developers are hunting for the tests, wondering if there are any tests for the feature they're working on and, if not, trying to figure out where to put their new tests.  Unfortunately, this disorganization is only the start of the problem.  Common Test Suite Problems  I've worked with many companies with large test suites and they tend to share some common problems. I list them in below in the order I try to address these problems (in other words, roughly easiest to hardest).   Tests often emit warnings  Tests often fail (\"oh, that sometimes fails. Ignore it.\")  There is little evidence of organization  Much of the testing code is duplicated  Testing fixtures are not used (or poorly used)  Code coverage is spotty  They take far too long to run   Problems are one thing, but what features do we want to see in large-scale test suites?   Tests should be very easy to write and run  They should run relatively quickly  The order in which tests run should not matter  Test output should be clean  It should be obvious where to find tests for a particular piece of code  Testing code should not be duplicated  Code coverage should be able to analyze different aspects of the system   Let's take a look at some of the problems and try to understand their impacts. While it's good to push a test suite into a desirable state, often this is risky if the underlying problems are ignored. I will offer recommendations for resolving each problem, but it's important to understand that these are recommendations . They may not apply to your situation.  Tests often emit warnings  This seems rather innocuous. Sure, code emits warnings and we're used to that. Unfortunately, we sometimes forget that warnings are warnings : there might very well be something wrong. In my time at the BBC, one of the first things I did was try to clean up all of the warnings. One was a normal warning about use of an undefined variable, but it was unclear to me from the code if this should be an acceptable condition. Another developer looked at it with me and realized that the variable should never be undefined: this warning was masking a very serious bug in the code, but the particular condition was not explicitly tested. By rigorously eliminating all warnings, we found it easier to make our code more correct, and in those places where things were dodgy, comments were inserted into the code to explain why warnings were suppressed. In short: the code became easier to maintain.  Another issue with warnings in the test suite is that they condition developers to ignore warnings. We get so used to them that we stop reading them, even if something serious is going on (on a related note, I often listen to developers complain about stack traces, but a careful reading of a stack trace will often reveal the exact cause of the exception). New warnings crop up, warnings change, but developers conditioned to ignore them often overlook serious issues with their code.  Recommendation : Eliminate all warnings from your test suite, but investigate each one to understand if it reflects a serious issue. Also, some tests will capture STDERR, effectively hiding warnings. Making warnings fatal while running tests can help to overcome this problem.  Tests often fail  For one client, their hour-long test suite had many failing tests. When I first started working on it, I had a developer walk me through all of the failures and explain why they failed and why they were hard to fix. Obviously this is a far more serious problem than warnings, but in the minds of the developers, they were under constant deadline pressures and as far as management was concerned, the test suite was a luxury to keep developers happy, not \"serious code.\" As a result, developers learned to recognize these failures and consoled themselves with the thought that they understood the underlying issues.  Of course, that's not really how it works. The developer explaining the test failures admitted that he didn't understand some of them and with longer test suites that routinely fail, more failures tend to crop up. Developers conditioned to accept failures tend not to notice them. They kick off the test suite, run and grab some coffee and later glance over results to see if they look reasonable (that's assuming they run all of the tests, something which often stops happening at this point). What's worse, continuous integration tools are often built to accomodate this. From the Jenkin's xUnit Plugin page :   Features   Records xUnit tests  Mark the build unstable or fail according to threshold values    In other words, there's an \"acceptable\" level of failure. What's the acceptable level of failure when you debit someone's credit card, or you're sending their medical records to someone, or you're writing embedded software that can't be easily updated?  Dogmatism aside, you can make a case for acceptable levels of test failure, but you need to understand the risks and be prepared to accept them. However, for the purposes of this document, we'll assume that the acceptable level of failure is zero.  If you absolutely cannot fix a particular failure, you should at least mark the test as TODO so that the test suite can pass. Not only does this help to guide you to a clean test suite, the TODO reason is generally embedded in the test, giving the next developer a clue what's going on.  Recommendation : Do not allow any failing tests. If tests fail which do not impact the correctness of the application (such as documentation or \"coding style\" tests), they should be separated from your regular tests in some manner and your systems should recognize that it's OK for them to fail.  There is little evidence of organization  As mentioned previously, a common lament amongst developers is the difficulty of finding tests for the code they're working on. Consider the case of HTML::TokeParser::Simple . The library is organized like this:  lib\/ └── HTML └── TokeParser ├── Simple │ ├── Token │ │ ├── Comment.pm │ │ ├── Declaration.pm │ │ ├── ProcessInstruction.pm │ │ ├── Tag │ │ │ ├── End.pm │ │ │ └── Start.pm │ │ ├── Tag.pm │ │ └── Text.pm │ └── Token.pm └── Simple.pm  There's a class in there named HTML::TokeParser::Simple::Token::ProcessInstruction . Where, in the following tests, would you find the tests for process instructions?  t\/ ├── constructor.t ├── get_tag.t ├── get_token.t ├── internals.t └── munge_html.t  You might think it's in the get_token.t test, but are you sure? And what's that strange munge_html.t test? Or the internals.t test? As mentioned, for a small library, this really isn't too bad. However, what if we reorganized our tests to reflect our library hierarchy?  t\/ └── tests\/ └── html\/ └── tokeparser\/ ├── simple\/ │ ├── token\/ │ │ ├── comment.t │ │ ├── declaration.t │ │ ├── tag\/ │ │ │ ├── end.t │ │ │ └── start.t │ │ ├── tag.t │ │ └── text.t │ └── token.t └── simple.t  It's clear that the tests for HTML::TokeParser::Simple::Token::Tag::Start are in t\/tests\/html\/tokeparser\/simple\/token\/tag\/start.t . And you can see easily that there is no file for processinstruction.t . This test organization not only makes it easy to find where your tests are, it's also easy to program your editor to automatically switch between the code and the tests for the code. For large test suites, this saves a huge amount of time. When I reorganized the test suite of the BBC's central metadata repository, PIPs , I followed a similar pattern and it made our life much easier.  ( Note : the comment about programming your editor is important. Effective use of your editor\/IDE is one of the most powerful tools in a developer's toolbox.)  Of course, your test suite could easily be more complicated and your top-level directories inside of your test directory may be structured differently:  t\/ ├── unit\/ ├── integration\/ ├── api\/ └── web\/  Recommendation : Organize your test files to have a predictable, discoverable structure. The test suite should be much easier to work with.  Much of the testing code is duplicated  We're aghast that that people routinely cut-n-paste their application code, but we don't even notice when people do this in their test code. More than once I've worked on a test suite with a significant logic change and I've had to find this duplicated code and either change it many places or try to refactor it so that it's in a single place and then change it. We already know why duplicated code is bad, I'm unsure why we tolerate this in test suites.  Much of my work in tests has been to reduce this duplication. For example, many test scripts list the same set of modules at the top. I did a heuristic analysis of tests on the CPAN and chose the most popular testing modules and that allowed me to change this:  use strict; use warnings; use Test::Exception; use Test::Differences; use Test::Deep; use Test::Warn; use Test::More tests =&gt; 42;  To this:  use Test::Most tests =&gt; 42;  You can easily use similar strategies to bundle up common testing modules into a single testing module that all of your tests use. Less boilerplate and you can easily dive into testing.  Or as a more egregious example, I often see something like this (a silly example just for illustration purposes):  set_up_some_data($id); my $object = Object-&gt;new($id); is $object-&gt;attr1, $expected1, 'attr1 works'; is $object-&gt;attr2, $expected2, 'attr2 works'; is $object-&gt;attr3, $expected3, 'attr3 works'; is $object-&gt;attr4, $expected4, 'attr4 works'; is $object-&gt;attr5, $expected5, 'attr5 works';  And then a few lines later:  set_up_some_data($new_id); $object = Object-&gt;new($new_id); is $object-&gt;attr1, $new_expected1, 'attr1 works'; is $object-&gt;attr2, $new_expected2, 'attr2 works'; is $object-&gt;attr3, $new_expected3, 'attr3 works'; is $object-&gt;attr4, $new_expected4, 'attr4 works'; is $object-&gt;attr5, $new_expected5, 'attr5 works';  And then a few lines later, the same thing ...  And in another test file, the same thing ...  Put that in its own test function and wrap those attribute tests in a loop. If this pattern is repeated in different test files, put it in a custom test library:  sub test_fetching_by_id ( $class, $id, $tests ) { my $object = $class-&gt;new($id); # this causes diagnostics to display the file and line number of the # caller on failure, rather than reporting *this* file and line number local $Test::Builder::Level = $Test::Builder::Level + 1; foreach my $test (@$tests) { my ( $attribute, $expected ) = @$test; is $object-&gt;$attribute, $expected, \"$attribute works for $class $id\"; } }  And then you call it like this:  my @id_tests = ( { id =&gt; $id, tests =&gt; [ [ attr1 =&gt; $expected1 ], [ attr2 =&gt; $expected2 ], [ attr3 =&gt; $expected3 ], [ attr4 =&gt; $expected4 ], [ attr5 =&gt; $expected5 ], ] }, { id =&gt; $new_id, tests =&gt; [ [ attr1 =&gt; $new_expected1 ], [ attr2 =&gt; $new_expected2 ], [ attr3 =&gt; $new_expected3 ], [ attr4 =&gt; $new_expected4 ], [ attr5 =&gt; $new_expected5 ], ] }, ); for my $test ( @id_tests ){ test_fetching_by_id( 'Object', $test-&gt;{id}, $tests-&gt;{test} ); }  This is a cleanly refactored data-driven approach. By not repeating yourself, if you need to test new attributes, you can just add an extra line to the data structures and the code remains the same. Or, if you need to change the logic, you only have one spot in your code where this is done. Once a developer understands the test_fetching_by_id() function, they can reuse this understanding in multiple places. Further, it makes it easier to find patterns in your code and any competent programmer is always on the lookout for patterns because those are signposts leading to cleaner designs.  Recommendation : Keep your test code as clean as your application code.  Testing fixtures are not used (or poorly used)  One difference between your application code and the test suite is in an application, we often have no idea what the data will be and we try to have a clean separation of data and code.  In your test suite, we also want a clean separation of data and code (in my experience, this is very hit-or-miss), but we often need to know the data we have. We set up data to run tests against to ensure that we can test various conditions. Can we give a customer a birthday discount if they were born February 29th? Can a customer with an overdue library book check out another? If our employee number is no longer in the database, is our code properly deleted, along with the backups and the git history erased? (kidding!)  When we set up the data for these known conditions under which to test, we call the data a test fixture . Test fixtures, when properly designed, allow us generate clean, understandable tests and make it easy to write tests for unusual conditions that may otherwise be hard to analyze.  There are several common anti-patterns I see in fixtures.   Hard to set up and use  Adding them to the database and not rolling them back  Loading all your test data at once with no granularity   In reviewing various fixture modules on the CPAN and for clients I have worked with, much of the above routinely holds true. On top of that, documentation is often rather sparse or non-existent. Here's a (pseudo-code) example of an almost undocumented fixture system for one client I worked with and it exemplified common issues in this area.  load_fixture( database =&gt; 'sales', client =&gt; $client_id, datasets =&gt; [qw\/ customers orders items order_items referrals \/], );  This had several problems, all of which could be easily corrected as code , but they built a test suite around these problems and had backed themselves into a corner, making their test suite dependent on bad behavior.  The business case is that my client had a product serving multiple customers and each customer would have multiple separate databases. In the above, client $client_id connects to their sales database and we load several test datasets and run tests against them. However, loading of data was not done in a transaction, meaning that there was no isolation between different test cases in the same process. More than once I caught issues where running an individual test case would often fail because it depended on data loaded by a different test case, but it wasn't always clear which test cases were coupled with which.  Another issue is that fixtures were not fine-tuned to address particular test cases. Instead, if you loaded \"customers\" or \"referrals\", you got all of them in the database. Do you need a database with a single customer with a single order and only one order item on it to test that obscure bug that occurs when a client first uses your software? There really wasn't any clean way of doing that; data was loaded in an \"all or nothing\" context. Even if you violated the paradigm and tried to create fine-tuned fixtures, it was very hard to write them due to the obscure, undocumented format needed to craft the data files for them.  Because transactions were not used and changes could not be rolled back, each *.t file would rebuild its own test database, a very slow process. Further, due to lack of documentation about the fixtures, it was often difficult to figure out which combination of fixtures to load to test a given feature. Part of this is simply due to the complex nature of the business rules, but the core issues stemmed from a poor understanding of fixtures. This client now has multiple large, slow test suites, spread across multiple repositories, all of which constantly tear down and set up databases and load large amounts of data. The test suites are both slow and fragile The time and expense to fix this problem is considerable due to how long they've pushed forward with this substandard setup.  What you generally want is the ability to easily create understandable fixtures which are loaded in a transaction, tests are run, and then changes are rolled back. The fixtures need to be fine-grained so you can tune them for a particular test case.  One attempt I've made to fix this situation is releasing DBIx::Class::EasyFixture , along with a tutorial . It does rely on DBIx::Class , the most popular ORM for Perl. This will likely make it unsuitable for some use cases.  Using them is very simple:  my $fixtures = DBIx::Class::EasyFixture-&gt;new(schema =&gt; $schema); $fixtures-&gt;load('customer_with_order_without_items'); # run your tests $fixtures-&gt;unload; # also unloads when out of scope  For the customer's code, we could satisfy the different database requirements by passing in different schemas. Other (well-documented) solutions, particularly those which are pure DBI based are welcome in this area.  Recommendation : Fine-grained, well-documented fixtures which are easy to create and easy to clean up.  Code coverage is poorly understood  Consider the following code:  float recip(float number) { return 1.0 \/ number; }  And a sample test:  assert recip(2.0) returns .5;  Congratulations! You now have 100% code coverage of that function.  For a statically typed language, I'm probably going to be moderately comfortable with that test. Alas, for dynamically typed languages we're fooling ourselves. An equivalent function in Perl will pass that test if we use recip(\"2 apples\") as the argument. And what happens if we pass a file handle? And would a Unicode number work? What happens if we pass no arguments? Perl is powerful and lets us write code quickly, but there's a price: it expects us to know what we're doing and passing unexpected kinds of data is a very common source of errors, but one that 100% code coverage will never (no pun intended) uncover. This can lead to false confidence.  To work around false confidence in your code, always assume that you write applications to create things and you write tests to destroy them. Testing is, and should be, an act of violence. If you're not breaking anything with your tests, you're probably doing it wrong.  Or what if you have that code in a huge test suite, but it's dead code? We tend to blindly run code coverage over our entire test suite, never considering whether or not we're testing dead code. This is because we slop our unit, integration, API and other tests all together.  Or consider the following test case:  sub test_forum : Tests(1) ($self) { my $site = $self-&gt;test_website; $site-&gt;login($user, $pass); $site-&gt;get('\/forum'); $site-&gt;follow_link( text =&gt; 'Off Topic' ); $site-&gt;post_ok({ title =&gt; 'What is this?', body =&gt; 'This is a test'. }, 'We should be able to post to the forum'); }  Devel::Cover doesn't know which code is test code and which is not. Devel::Cover merely tells you if your application code was exercised in your tests. You can annotate your code with \"uncoverable\" directives to tell Devel::Cover to ignore the following code, but that potentially means sprinkling your code with annotations all over the place.  There are multiple strategies to deal with this. One of the simplest is to merely run your code coverage tools over the public-facing portions of your code, such as web or API tests. If you find uncovered code, you either have code that is not fully tested (in the sense that you don't know if your API can really use that code) or, if you cannot write an API test to reach that code, investigate if it is dead code.  You can do this by grouping your tests into subdirectories:  t\/ |--api\/ |--integration\/ `--unit\/  Alternatively, if you use Test::Class::Moose , you can tag your tests and only run coverage over tests including the tags you wish to test:  My::Test::Class::Moose-&gt;new({ include_tags =&gt; [qw\/api\/], })-&gt;runtests;  If you start tagging your tests by the subsystems they are testing, you can then start running code coverage on specific subsystems to determine which ones are poorly tested.  Recommendation : Run coverage over public-facing code and on different subsystems to find poor coverage.  They take far too long to run  The problem with long-running test suites is well known, but it's worth covering this again here. These are problems that others have discussed and that I have also personally experienced many times.   With apologies to XKCD  In the best case scenario for developers who always run that long-running test suite, expensive developer time is wasted while the test suite is running. When they launch that hour-long (or more) test suite, they frequently take a break, talk to (read: interrupt) other developers, check their Facebook, or do any number of things which equate to \"not writing software.\" Yes, some of those things involve meetings or research, but meetings don't conveniently schedule themselves when we run tests and for mature products (those which are more likely to have long-running test suites), there's often not that much research we really need to do.  Here are some of the issues with long-running test suites:   Expensive developer time is wasted while the test suite runs  Developers often don't run the entire test suite  Expensive code coverage is not generated as a result  Code is fragile as a result   What I find particularly curious is that we accept this state of affairs. Even a back-of-the-envelope calculation can quickly show significant productivity benefits that will pay off in the long run by taking care of our test suite. I once reduced a BBC test suite's run time from one hour and twenty minutes down to twelve minutes ( Note: today I use a saner approach that results in similar or greater performance benefits ). We had six developers on that team. When the test suite took over an hour to run, they often didn't run the test suite. They would run tests on their section of code and push their code when they were comfortable with it. This led to other developers finding buggy code and wasting time trying to figure out how they broken it when, in fact, someone else broke the code.  But let's assume each developer was running the test suite at least once a day (I'm careful about testing and often ran mine twice a day). By cutting test suite run time by over an hour, we reclaimed a full day of developer productivity every day! Even if it takes a developer a month to increase perfomance by that amount it pays for itself many times over very quickly. Why would you not do this? As a business owner, wouldn't you want your developers to save time on their test suite so they can create features faster for you?  There are several reasons why this is difficult. Tasking a developer with a block of time to speed up a test suite means the developer is not creating user-visible features during that time. For larger test suites, it's often impossible to know in advance just how much time you can save or how long it will take you to reach your goal. In most companies I've worked with, the people who can make the decision to speed up the test suite are often not the people feeling the pain. Productivity and quality decrease slowly over time, leading to the boiling frog problem .  What's worse: in order to speed up your test suite without affecting behavior, the test suite often has to be \"fixed\" (eliminating warnings, failures, and reducing duplication) to ensure that no behavior has been changed during the refactor.  Finally, some developers simply don't have the background necessary to implement performance optimizations. While performance profiles such as Perl's Devel::NYTProf can easily point out problem areas in the code, it's not always clear how to overcome the discovered limitations.  The single biggest factor in poor test suite performance for applications is frequently I\/O. In particular, working with the database tends to be a bottleneck and there's only so much database tuning that can be done. After you've profiled your SQL and optimized it, several database-related optimizations which can be considered are:   Using transactions to clean up your database rather than rebuilding the database  Only connect to the database once per test suite (hard when you're using a separate process per test file)  If you must rebuild the database, maintain a pool of test databases and assign them as needed, rebuilding used ones in the background  Use smaller database fixtures instead of loading everything at once   After you've done all you can to improve your database access, you may find that your test suite is \"fast enough\", but if you wish to go further, there are several steps you can take.  Recommendations  Aggressively search for and remove duplicated tests.  For poorly organized test suites, developers sometimes make the mistake of putting tests for something in a new *.t file or add them to a different *.t file, even if related tests already exist. This strategy can be time-consuming and often does not result in quick wins.  Use code coverage aggressively  For one test suite, I found that we were using a pure Perl implementation of JSON. As the test suite used JSON extensively, switching to JSON::XS gave us a nice performance boost. We may not have noticed that if we hadn't been profiling our code.  Look for code with \"global\" effects  On one test suite, I ensured that Universal::isa and Universal::can cannot be loaded. It was a quick fix and sped up the test suite by 2% (several small accumulations of improvements can add up quickly).  Inline \"hot\" functions.  Consider the following code which runs in about 3.2 seconds on my computer:  #!\/usr\/bin\/env perl use strict; use warnings; no warnings 'recursion'; for my $i ( 1 .. 40 ) { for my $j ( 1 .. $i**2 ) { my $y = factorial($j); } } sub factorial { my $num = shift; return 1 if $num &lt;= 1; return $num * factorial($num - 1); }  By rewriting the recursive function as a loop, the code takes about .87 seconds:  sub factorial { my $num = shift; return 1 if $num &lt;= 1; $num *= $_ for 2 .. $num - 1; return $num; }  By inlining the calculation, the code completes in .69 seconds:  for my $i ( 1 .. 40 ) { for my $j ( 1 .. $i**2 ) { my $y = $j; if ( $y &gt; 1 ) { $y *= $_ for 2 .. $y - 1; } } }  In other words, in our trivial example, the inlined behavior is roughly 20% faster than the iterative function and 80% faster than the recursive function.  Recompile your Perl  You may wish to recompile your Perl to gain a performance improvement. Many Linux distributions ship with a threaded Perl by default. Depending on the version of Perl you ship with, you can gain performance improvements of up to 30% by recompiling without threads. Of course, if you use threads, you'll feel very stupid for doing this. However, if you don't make heavy use of threads, switching to a forking model for the threaded code may make the recompile worth it. Naturally, you'll need to heavily benchmark your code (preferably under production-like loads) to understand the trade-offs here.  Preload modules  If your codebase makes heavy use of modules that are slow to load, such as Moose , Catalyst , DBIx::Class and others, preloading them might help. forkprove is a utility written by Tatsuhiko Miyagawa that allows you to preload slow-loading modules and then forks off multiple processes to run your tests. Using this tool, I reduced one sample test suite's run time from 12 minutes to about a minute . Unfortunately, forkprove doesn't allow schedules, a key component often needed for larger test suites. I'll explain that in the next section.  Parallel tests  Running tests in parallel is tricky. Some tests simply can't be run with other tests. Usually these are tests which alter global state in some manner that other processes will pick up, or might cause resource starvation of some kind.  Or some tests can be run in parallel with other tests, but if several tests are updating the same records in the database at the same time, locking behavior might slow down the tests considerably.  Or maybe you're running 4 jobs, but all of your slowest tests are grouped in the same job: not good.  To deal with this, you can create a schedule that assigns different tests to different jobs, based on a set of criteria, and then puts tests which cannot run in parallel in a single job that runs after the others have completed.  You can use TAP::Parser::Scheduler to create an effective parallel testing setup. You can use this with TAP::Parser::Multiplexer to create your parallel tests. Unfortunately, as of this writing there's a bug in the Multiplexer whereby it uses select in a loop to read the parser output. If one parser blocks, none of the other output is read. Further, the schedule must be created prior to loading your test code, meaning that if your tests would prefer a different schedule, you're out of luck. Also, make test currently doesn't handle this well. There is work being done by David Golden to alleviate this problem.  My preferred solution is to use Test::Class::Moose . That has built-in parallel testing and writing schedules is very easy. Further, different test cases can simply use a Tags(noparallel) attribute to ensure that they're run sequentially after the parallel tests.  Aside from the regular benefits of Test::Class::Moose , an interesting benefit of this module is that it loads all of your test and application code into a single process and then forks off subprocesses. As a result, your code is loaded once and only once. Alternate strategies which try to fork before loading your code might still cause the code to be loaded multiple times.  I have used this strategy to reduce a 12 minute test suite to 30 seconds .  Distributed tests  Though I haven't used this module, Alex Vandiver has written TAP::Harness::Remote . This module allows you to rsync directory trees to multiple servers and run tests on those servers. Obviously, this requires multiple servers.  If you want to roll your own version of this, I've also released TAP::Stream , a module that allows you to take streams (the text, actually) of TAP from multiple sources and combine them into a single TAP document.  Devel::CoverX::Covered  There is yet another interesting strategy: only run tests that exercise the code that you're changing. Johan Lindström wrote Devel::CoverX::Covered . This modules is used in conjunction with Paul Johnson's Devel::Cover to identify all the places in your tests which cover a particular piece of code. In the past, I've written tools for vim to read this data and only run relevant tests. This is a generally useful approach, but there are a couple of pitfalls.  First, if you test suite takes a long time to run, it will take much, much longer to run with Devel::Cover . As a result, I recommend that this be used with a special nightly \"cover build\" and have the results synched back to the developers.  Second, when changing code, it's easy to change which tests cover your code, leading to times when this technique won't cover your actual changes thoroughly. In practice, this hasn't been a problem for me, but I've not used it enough to say that with confidence.  Recommendation : Don't settle for slow test suites. Pick a goal and work to achieving that goal (it's easy to keep optimizing for too long and start getting diminishing marginal returns).  Test::Class::Moose  If you start creating a large Web site, do you start writing a bunch of individual scripts, each designed to handle one URL and each handling their own database access and printing their output directly to STDOUT? Of course not. Today, professional developers reach for Sinatra, Seaside, Catalyst, Ruby on Rails or other Web frameworks. They take a bit more time to set up and configure, but we know they generally save more time in the long run. Why wouldn't you do that with your test suite?  If you're using Perl, many of the problems listed in this document can be avoided by switching to Test::Class::Moose . This is a testing framework I designed to make it very easy to test applications. Once you understand it, it's actually easy to use for testing libraries, but it really shines for application testing.  Note that I now regret putting Moose in the name. Test::Class::Moose is a rewrite of Test::Class using Moose , but it's not limited to testing Moose applications. It uses Moose because internally it relies on the Moose meta-object protocol for introspection.  Out of the box you get:   Reporting  Parallel tests (which optionally accepts a custom schedule)  Tagging tests (slice and dice your test suite!)  Test inheritance (xUnit for the win!)  Full Moose support  Test control methods (startup, setup, teardown, shutdown)  Extensibility  All the testing functions and behavior from Test::Most   To learn about xUnit testing in Perl, you may wish to read a five-part tutorial I published at Modern Perl Books:   Organizing test suites with Test::Class  Reusing test code  Making your testing life easier  Using test control methods  Working with Test::Class test suites   That tutorial is slightly out of date (I wrote it a few years ago), but it explains effective use of Test::Class and some common anti-patterns when using it.  Doug Bell has started a tutorial for Test::Class::Moose . That also needs updating, but between those and reading the Test::Class::Moose documentation, you should be able to get up to speed fairly quickly.  ","title":"The Zen of Test Suites","url":"\/articles\/zen-of-test-suites.html"},{"body":"    Consistency Is Your Friend  Language Design  Who the Hell is Kim?  This Ain’t Kim  Turning Corinna into Kim  Conclusion     Consistency Is Your Friend  Note : None of the following is set in stone. This is merely a way to start discussion on some issues with syntactic inconsistencies in Corinna.  Years ago I was working with some graphing software where points would be declared via:  var point = point(2,3);  Except that the point function took Y,X as arguments, not the near universally-accepted X,Y . To this day, I loathe the author of that library for making my life hell.  On a more personal note, while building Tau Station  —a narrative sci-fi MMORPG—I was trying to respect proper separation of concerns and not building god objects, so when it came time to check if an NPC has a mission for a character, I had this:  if ( $missions-&gt;has_mission_for( $character, $npc ) ) { ... }  Or was it this?  if ( $missions-&gt;has_mission_for( $npc, $character ) ) { ... }  To this day, I cannot remember which style I used and what justification I had for that, but it was a constant source of bugs because I kept getting the $npc and $character swapped. So I bit the bullet and decided that, as much as possible, Tau Station code would follow SVO (subject-verb-object) syntax, instead of the weird “context-verb-subject-object” I had created. Or was it “context-verb-object-subject”? Who knows? Instead, the above became this:  if ( $npc-&gt;has_missions_for($character) ) { ... }  And now the meaning of the code is crystal clear, though people will complain about “God objects” ( which I address here  ).  Consistency brings clarity. Reasoning about code is hard enough when it appears to be an ad-hoc mishmash of rules.    So, um, that brings us to the Corinna project  , my attempt to bring modern object-oriented programming to the Perl language.  Language Design  As I’ve previously pointed out, my first foray into language design didn’t go as well as I hoped. At its heart, you can think of language design as consisting of three parts:   Semantics  Syntax  Community   I mostly had the semantics and syntax down, but I hadn’t done a great job of paying attention to the community. The community feedback was mixed , to put it politely. So an IRC channel was created (irc.perl.org #cor) and a github project  was started and anyone with an interest in the project was welcome to share their thoughts.  By including the community, we’ve done a fantastic job of turning Corinna from an interesting sideshow to something that is (mostly) popular with the Perl community. If we can bring her over the finish line, we have something that is likely to be included in the Perl core as an experimental project. You have to ask for it by name:  use feature 'class'; class Foo { ... }  And then you get all the lovely OO goodies in a powerful, declarative syntax.  We’re soooooo close to being able to propose an RFC that I am loathe to throw a wrench in the works, but I must. I must because Kim is very, very unhappy with Corinna.  Who the Hell is Kim?  Please note that much of this is largely derived from emails I’ve swapped with Damian Conway  . I’ve rewritten it considerably, but the core of this is him noting inconsistencies that have been discussed on the IRC channel, but not put together conherently. Corinna has evolved with some serious inconsistencies and the time to consider fixing them is now, not after it’s released.  Corinna, as some of you may know, was chosen as the name of this project because my nom de guerre ,   is “Ovid” and he wrote poems to Corinna, a love interest (if you’re familiar with the poetry, you know I’m keeping this family-friendly). Kim isn’t jealous of Corinna. She’s furious because Corinna, after literally years of design, is breaking some cardinal rules of consistency. So, let’s dig into that.  So let’s look at the way we declare things in Perl. In particular, we’ll look at my variables, subroutines, and packages.  my $answer = 42; sub douglas_adams { ... } package Restaurant { ... }  Those all seem different, but we start declaring what type of thing we have: a variable, a subroutine, or a package. Then we follow that up with the name of the thing, followed by the optional set up of that thing.  But what if we want to modify how those things behave? Attributes have been alternately proposed and rejected by many developers, but if you use threads::shared in Perl, you can declare a variable as shared across threads:  my $answer :shared = 42;  What if you want to declare a subroutine as an lvalue?  sub douglas_adams :lvalue {...}  Or declaratively state the version of a package without using the procedural $VERSION assignment?  package Restaurant v3.1.4 { ... }  In fact, if we continue down this road, we see a pattern start to emerge:         keyword  identifier  modifiers?  setup?      my  $lexvar  :shared      our  $packvar  :Tracked  = 0    state  $statevar  :Readonly  = 1    sub  action  :lvalue  () {...}    package  Namespace  v1.2.3  {...}    format  Report    = ... .     KIM stands for “Keyword”, “Identifier”, and “Modifier” (yes, it should be “KIMS”, but I like the “Kim versus Corinna” description. Sue me.)  Kim likes consistency. You must always declare the kind of thing you’re going to use and then name it. You can then optionally modify its base behavior and then optionally set it up. Very, very consistent:  KEYWORD IDENTIFIER MODIFIERS SETUP  Of course, you can point to plenty of areas where Perl is not consistent, but Corinna is designed to improve the Perl language, not add to the inconsistency, so let’s look at a few examples:   KIM features in Corinna        keyword  identifier  modifiers?  setup?      role  Tracked    {...}      class  Root    {...}      slot  $msg  :param      slot  $handler  :handles(exists delete)  = Handler->new;    slot  $size  :reader :param  = 100;    slot  $created  :reader  = time;      method  is_root ()    {...}    method  obj_count  :common ()  {...}    method  insert  :private ($key,$value)  {...}    method  show ($msg)    {...}    method  obj_count  :overrides :common ()  {...}     So that’s actually looking pretty good. Corinna is developing with a clear, consistent KIM syntax. But if that was all, this wouldn’t be a helpful article. That’s the good; it’s time to look at the bad. And maybe the ugly.  This Ain’t Kim  Let’s look at a few examples of inconsistencies. We’ll start with putting the modifier first.   Modifiers First        modifier  keyword  identifier  setup      abstract  class  Counter  {...}     And let’s mix up keywords and modifiers.   Keywords as Modifiers        keyword  identifier  keyword as modifiers  setup      class  Handler  isa Counter does Tracked  {...}     Let’s use modifers plus keywords plus identifiers plus modifiers plus ...   Keywords gonna Keyword         modifier  keyword  identifier  modifiers  setup      before  method  obj_count  :overrides :common ()  { warn “Counting...“; }    after  method  obj_count  :overrides :common ()  { warn “...done”; }    around  method  obj_count  :overrides :common ()  { return 1 + $self->$ORIG(); }     So, yeah. Damian pointed all of this out to me and he’s right. Corinna is close , but there are syntactical inconsistencies which, if agreed that they should be addressed, should be addressed before an RFC, not after.  Turning Corinna into Kim  The name’s still Corinna, but she needs KIM.  When I started converting Tau Station code to (in)consistently use SVO syntax, it made it much easier to maintain. Doing the same thing with Corinna is worthwhile.  So, how do we address all these inconsistencies? Corinna semantics are OK. We just need some tweaks to the syntax.         keyword  identifier  modifiers?  setup?      class  Counter  :abstract  {...}    class  Handler  :isa(Counter) :does(Tracked)  {...}    method  obj_count  :before :overrides :common  () { warn “Counting...“; }    method  obj_count  :after :overrides :common  () { warn “...done”; }    method  obj_count  :around :overrides :common  () { return 1 + $self->$ORIG(); }     And here’s a slightly modified example from Damian showing this more consistent syntax:  role Tracked { slot $msg :param; method report () { $self-&gt;show($msg++) } method show; # required } class Root { method is_root () { return 1; } } class Counter :abstract { slot $obj_count :common :reader = 0; ADJUST { $obj_count++ } DESTRUCT { $obj_count-- } } class Handler :isa(Counter) :does(Tracked) { slot $handler :handles(exists delete) = Handler-&gt;new; slot $size :reader :param = 100; slot $created :reader = time; ADJUST { croak(&quot;Too small&quot;) if $size &lt; 1; } DESTRUCT { $handler-&gt;shutdown; } method insert :private ($key, $value ) { if ( ! $self-&gt;exists($key) ) { $handler-&gt;set( $key, $value ); } } method show ($msg) { say $msg; } method obj_count :common :overrides () { $self-&gt;next::method() - 42; } method obj_count :common :before () { warn &quot;Counting...&quot;; } method obj_count :common :around () { return 1 + $self-&gt;$ORIG(); } method obj_count :common :after () { warn &quot;...done&quot;; } }  Conclusion  As you can see from the sample code, this still looks like Corinna. We have the semantics and syntax, but do we have the community? The changes aren’t huge, but they might prove controversial because Corinna would again be changing. But I think it’s changing for the better. Further, by establishing the KIM principle now, further upgrades to the Perl language can have guidance on maintaining consistency.  ","title":"Language Design Consistency","url":"\/articles\/language-design-consistency.html"},{"body":" If you search LinkedIn for groups related to Scrum, you get almost 700 results as of this writing. Many of these are fantastic groups, with great people offering advice from the trenches. If you use Agile, particularly Scrum, you should follow these groups. Just beware that there is a problem there: sometimes you find yourself staring into the wide eyes of the converted. It's particularly bad if you're staring in a mirror.  From time to time people in various Scrum groups ask \"What are the weakness of Scrum?\" And, I kid you not, more than once I have seen the reply \"none, Scrum is perfect.\" Those respondents are the people who won't change their mind and won't change the subject and I very much want to gently shove them into a meeting full of their fellow converted and go back to getting work done. Out of deference to their disability, I will set a goal for the meeting, time-box it at eight hours, and when the participants email me the action items, I will print out that email and burn it. It will be a good day.  Or, as a fellow Agile trainer wrote to me \"some of my clients are dumping Scrum just to get away from the zealots.\"  Lord, save me from your followers.  Fortunately, those who believe that Scrum (or XP, or Kanban, or Scrumban, or, or, or ...) is perfect, are in the minority. They're the obvious crackpots that we greet with a fixed smile and back away from slowly, but there are the less obvious crackpots who are far more dangerous because they sound so reasonable.  These are the ones who believe with a fiery passion that everything should be Agile. These people are going to hurt your company.  To understand this, we need to remember what Agile is. Agile development is simply a set of practices designed to better cope with uncertainty and change. That's all. A key indicator of when to use Agile is when you're developing a new technology and saying \"wow, look at all the possibilities!\" When you're breaking new ground, when you're pushing change rather than following the herd, then Agile is a great approach.  But what if you don't have a rapidly changing environment? What if quality and reproducibility are more important considerations? Janitors don't need product owners. They don't create \"who\/what\/why\" story cards. They don't have sprints. If your janitorial staff are struggling with a chaotic, uncertain environment, you have bigger problems on your hands. Mind you, there are janitors who have to deal with rapidly changing environments and uncertainty. We call them SEAL teams.  OK, so maybe you've outsourced your janitorial staff and are thinking everything else can be Agile. Let's head over to your accounting department. Forget about your prejudices for a moment and look at what accounting does. Ask yourself one question \"do I want them using a project management system that's optimized for uncertainty and change, or quality and reproducibility?\" Remember, your answer not only has financial implications, but legal ones as well.  When it's phrased in those terms, the answer is fairly clear. Even the most ardent Agile proponents would have to agree that an accounting team may need to be run differently from a software development team. There is no \"one size fits all\" suit for project management and if you're trying to cram everything in your organization into the same system, you're begging for trouble.  As a rule of thumb, for every department ask yourself how \"new\" their product is. The newer it is, the more likely it's being born in an uncertain environment and Agile is a good approach. If it's an older, mature, stable environment, consider PRINCE2, TQM, or other related project management solutions. Lean solutions fit somewhere in the middle of those two extremes.  Don't let yourself get caught up in the hype. Agile solutions are excellent, but don't go out in pairs, knocking on doors and preaching to the masses. Different problems require different solutions and we need to give up our ever present quest to find the One True Way.  If you'd like to know more, you might want to read when to choose agile  ","title":"When Going Agile Can Hurt Your Company","url":"\/articles\/going-agile-can-hurt-your-company.html"},{"body":" Yeah, sit back in your chair, you silly, naïve developer. I've been building software since before you were potty-trained and I know what I'm doing. I know what customers want and I know how to build it for them.  Er, except for the fact that I'm usually wrong.  When I first encountered A\/B testing, I had fixed a bug in a search engine and tried to push my change. It was rejected. Why? Because I hadn't wrapped it in an A\/B test. I was confused. Why would I want to A\/B test a bugfix? Under certain conditions our search engine would return no results and I fixed it to ensure that it would not only return results, but they would be relevant results.  Boss: I don't care. A\/B test it anyway.  So I did. And it was painful, waiting days for the results to come in ... and being dismayed to see very clearly that improving the behavior of the search engine led to significantly lower customer conversion. I even questioned the validity of the data, but to no avail. It took me a long time to truly realize the implications, but it harmonizes quite well with other things I teach about software:  Software behavior should be considered buggy when it causes unwanted customer behavior, not unwanted software behavior.  Yes, that's an oversimplification, but bear with me.  A\/B Testing: Slayer of Egos  I had a client who introduced a horizontal scroll bar to their e-commerce site and significantly improved sales. Another company found that removing social media buttons increased conversions . And I found that our customers didn't respond well to better search results.  For many people those ideas might sound counter-intuitive, but they had real data to back them up. Instead, here's what often happens when you have experts guiding development by their \"experience\":   We need multiple pictures of our product on the \"Buy now\" page! Sales drop.  We need to show related items so we can cross-sell! Sales drop.  We need to show more relevant search results! Sales drop.   Quite often you won't find out that sales dropped as a result of a particular change because you haven't measured your customer's behavior and ignoring your customers is, by anyone's reckoning, a recipe for disaster. Remember how Digg went from king of the web to court jester in 2010 because they ignored their customers?  A\/B testing is not a silver bullet, but it's an excellent way of getting real, honest, data-driven answers to your questions. And if you have an ego, it's a punch in the gut. Again and again and again. Why? Because the vast majority of A\/B tests are inconclusive (no apparent change in behavior) or negative (you probably lost money). Your customers don't care about how experienced you are.  The Customer Isn't Always Right  Your customers care about what your customers care about. Unfortunately, they often don't know what they care about until they experience it first-hand or some bandwagon effect kicks in and everyone piles on (this can be either good or bad for you).  To give an example of how problematic this can be, consider the case of a company in France a friend of mind was running (sorry, I can't disclose the name). It was was a very innovative paid search engine covering a very technical field. The company raised a lot of seed funding and was growing steadily, but it wasn't profitable because potential customers weren't signing up fast enough. Surveys revealed they wanted the search engine to return research results in French and English. This was because the top research in this field was published in English and the French professionals wanted access to it.  Given that the search engine was in French and it was searching highly technical documents, converting it to also search English documents, search them correctly , buy subscriptions to the various technical sources those documents were found in, and presenting them seamlessly on a French-language website turned out to be both expensive and time-consuming. And it wasn't just development costs: it was ongoing costs to subscribe to, parse, and index the English language materials.  The feature was released to much fanfare and potential customers didn't care. At all. They asked for this feature but it had no significant impact on conversion. As it turns out, French professionals may very well speak English, but they still preferred French. What they wanted wasn't English-language documents; they wanted a reassurance they were getting a good value for their money . Given that the company had a simple metric&mdash;new customer subscriptions&mdash;A\/B testing could have been a far less expensive way of divining this information. Maybe only provide a single English-language source. Maybe provide testimonials. Maybe something as simple as playing with language or images may have had an impact?  Instead, the cost of development, the opportunity cost of not developing other features or spending on marketing, and the ongoing cost of maintaining the English-language corpus were all significant contributing factors to the collapse of the company.  The literature is rife with stories of companies listening to what their customers say instead of paying attention to what their customers do. A\/B testing tells you what they're actually doing.  Limitations of A\/B Testing  A\/B testing is a powerful tool in your toolbox, but it shouldn't be considered the only tool. Further, just as you don't use a hammer with screws, you shouldn't misuse A\/B testing. Evan Miller has a great article entitled How Not To Run an A\/B Test .  But statistical flaws aside, there are other issues with A\/B testing. As part of your agile toolkit, you want to run them often . If you only release new versions of your software quarterly you're going to struggle to rapidly respond to customer needs. If you're testing at the bottom of a conversion funnel and only get 10 visitors a week, you might wait months to get a meaningful result. Or maybe you've run a very successful A\/B test, but it's not related to KPIs of the company.  And the killer, one I've seen all too often: \"that idea is rubbish! We don't need to test that!\"  If you've made A\/B tests painful to set up and can only run them periodically, I can understand that attitude (but the A\/B test isn't the real problem there). However, if you can easily run tests and respond to results quickly, it often makes sense to test \"dumb\" ideas. Consider the case of the horizontal scroll bar I mentioned earlier.  You'll have plenty of experts telling you why horizontal scroll bars are a disaster , so why did it work for my aforementioned client?  First, the key thing to remember is that A\/B tests tell you what your customers are doing, but not why.  With the horizontal scroll bar, the test clearly showed increased conversion, but the designer was extremely unhappy. After a lot of thought and examining the page, she noticed something interesting. A side effect of the horizontal scroll bar was that the full product description was now visible without vertical scrolling. She redesigned the page to use a vertical scroll bar instead of a horizontal one, but kept the full product description visible.  Once again there was another nice increase in conversion rates, significantly better than the original version.  Your expertise isn't in dictating features, it's in designing experiments and interpreting the results.  So you see? Your experience still matters and you can keep feeding your ego, but now you have hard data to back it up.  Summary  If you're not doing A\/B testing, you really should consider it. There are plenty of companies which provide simple tools for integrating A\/B testing into your Web site. Once you understand them, and you get a feel for the power of A\/B testing, it's time for you to start building an internal tool that's more suitable for your needs. You'll be able to test things you could never test before, and you'll have better access to customer segmentation data.  Stop doing ego-driven development. This is a typical story card we see today:  As a &lt;developer&gt; I want to &lt;build random stuff&gt; So that &lt;the boss stays happy&gt;  This is what you're really looking for:  We suspect that &lt;building this feature&gt; ... &lt;for these people&gt; ... will achieve &lt;this measurable result&gt; We will know we succeeded when we see &lt;this market signal&gt;.  ","title":"The Surprises of A\/B Testing","url":"\/articles\/the-surprises-of-ab-testing.html"},{"body":"  Immutable Objects  I’ve been spending time designing Corinna  , a new object system to be shipped with the Perl language. Amongst its many features, it’s designed to make it easier to create immutable objects, but not everyone is happy with that. For example, consider the following class:  class Box { has ($height, $width, $depth) :reader :new; has $volume :reader = $width * $height * $depth; } my $original_box = Box-&gt;new(height=&gt;1, width=&gt;2, depth=&gt;3); my $updated_box = $original_box-&gt;clone(depth=&gt;9); # h=1, w=2, d=9  Because none of the slots have a :writer attribute, there is no way to mutate this object. Instead you call a clone method, supplying an overriding value for the constructor argument you need to change. The $volume argument doesn’t get copied over because it’s derived from the constructor arguments.  But not everyone is happy with this approach  . Aside from arguments about utility of the clone method, the notion that objects should be immutable by default has frustrated some developers reading the Corinna proposal. Even when I point out just adding a :writer attribute is all you need to do to get your mutability, people still object. So let’s have a brief discussion about immutability and why it’s useful.  But first, here’s my last 2020 Perl Conference presentation on Corinna.     The Problem  Imagine, for example, that you have a very simple Customer object:  my $customer = Customer-&gt;new( name =&gt; &quot;Ovid&quot;, birthdate =&gt; DateTime-&gt;new( ... ), );  In the code above, we’ll assume the $customer can give us useful information about the state of that object. For example, we have a section of code guarded by a check to see if they are old enough to drink alcohol:  if ( $ovid-&gt;old_enough_to_drink_alcohol ) { ... }  The above looks innocent enough and it’s the sort of thing we regularly see in code. But then this happens:  if ( $ovid-&gt;old_enough_to_drink_alcohol ) { my $date = $ovid-&gt;birthdate; ... # deep in the bowels of your code my $cutoff_date = $date-&gt;set( year =&gt; $last_year ); # oops! ... }  We had a guard to ensure that this code would not be executed if the customer wasn’t old enough to drink, but now in the middle of that code, due to how DateTime is designed, someone’s set the customer birth date to last year! The code, at this point, is probably in an invalid state and its behavior can no longer be considered correct.  But clearly no one would do something so silly, would they?  Global State  We’ve known about the dangers of global state  for a long time. For example, if I call the following subroutine, will the program halt or not?  sub next ($number) { if ( $ENV{BLESS_ME_LARRY_FOR_I_HAVE_SINNED} ) { die &quot;This was a bad idea.&quot;; } return $number++; }  You literally cannot inspect the above code and tell me if it will die when called because you cannot know, by inspection, what the BLESS_ME_LARRY_FOR_I_HAVE_SINNED environment variable is set to. This is one of the reasons why global environment variables are discouraged.  But here we’re talking about mutable state. You don’t want the above code to die, so you do this:  $ENV{BLESS_ME_LARRY_FOR_I_HAVE_SINNED} = 0; say next(4);  Except that now you’ve altered that mutable state and anything else which relies on that environment variable being set is unpredicatable. So we need to use local to safely change that in the local scope:  { local $ENV{BLESS_ME_LARRY_FOR_I_HAVE_SINNED} = 0; say next(4); }  Even that is not good because there’s no indication of why we’re doing this   , but at least you can see how we can safely change that global variable in our local scope.  ORMs  And I can hear your objection now:   “But Ovid, the DateTime object in your first example isn’t global!”   That’s true. What we had was this:  if ( $ovid-&gt;old_enough_to_drink_alcohol ) { my $date = $ovid-&gt;birthdate; ... # deep in the bowels of your code my $cutoff_date = $date-&gt;set( year =&gt; $last_year ); # oops! ... }  But the offending line should have been this:  # note the clone(). my $cutoff_date = $date-&gt;clone-&gt;set( year =&gt; $last_year );  This is because the set method mutates the object in place, causing everything holding a reference to that object to silently change . It’s not global in the normal sense, but this action at a distance is a source of very real bugs  .  It’s a serious enough problem that DateTime::Moonpig  and DateTimeX::Immutable  have both been written to provide immutable DateTime objects, and that brings me to DBIx::Class  , an excellent ORM for Perl.  As of this writing, it’s been around for about 15 years and provides a component called DBIx::Class::InflateColumn::DateTime  . This allows you to do things like this:  package Event; use base 'DBIx::Class::Core'; __PACKAGE__-&gt;load_components(qw\/InflateColumn::DateTime\/); __PACKAGE__-&gt;add_columns( starts_when =&gt; { data_type =&gt; 'datetime' } create_date =&gt; { data_type =&gt; 'date' } );  Now, whenever you call starts_when or create_date on an Event instance, you’ll get a DateTime object instead of just the raw string from the database. Further, you can set a DateTime object and not worry about your particular database’s date syntax. It just works .  Except that the object is mutable and we don’t want that.   You can fix this by writing your own DBIx::Class component to use immutable DateTime objects.  package My::Schema::Component::ImmutableDateTime; use DateTimeX::Immutable; use parent 'DBIx::Class::InflateColumn::DateTime'; sub _post_inflate_datetime { my ( $self, @args ) = @_; my $dt = $self-&gt;next::method(@args); return DateTimeX::Immutable-&gt;from_object( object =&gt; $dt ); } 1;  And then load this component:  __PACKAGE__-&gt;load_components( qw\/+My::Schema::Component::ImmutableDateTime\/ );  And now, when you fetch your objects from the database, you get nice, immutable DateTime s. And it will be interesting to see where your codebase fails!  Does all of this mean we should never use mutable objects? Of course not. Imagine creating an immutable cache where, if you wanted to add or delete an entry, you had to clone the entire cache to set the new state. That would likely defeat the main purpose of a cache: speeding things up. But in general, immutability is a good thing and is something to strive for. Trying to debug why code far, far away from your code has reset your data is not fun.  ","title":"Why Do We Want Immutable Objects?","url":"\/articles\/using-immutable-datetime-objects-with-dbixclass.html"},{"body":"    Introduction  CamelCaseNames or underscore_names?  Plural or Singular Tables?  Don’t name the ID column “id”  Column Naming  Avoid NULL Values  Database Types  What does a NULL value mean?  NULLs lead to logical impossibilities  Summary     Introduction  When moving from project to project, it’s unfortunate that we find that there are no consistent standards on database design, despite SQL having been around for decades. I suspect that much of this is because most developers don’t understand database design . In fact, with my years of hiring developers, only a handful of times have I met developers who can properly normalize a database. To be fair, normalization can be hard, but most developers I’ve interviewed, even excellent ones with strong SQL skills, don’t have database design skills.  But this article isn’t about database normalization. If you want to learn more, here’s a short talk I gave which explains the basics.     Instead, when you have a working database, the question we want to know is “what standards can we apply which make it easier to use that database?” If these standards were to be widely adopted, databases would be easier to use because you wouldn’t have to learn and remember a new set of standards every time you work with a new database.  CamelCaseNames or underscore_names?  Let’s get this out of the way quickly. I routinely see database examples online where we see table names like CustomerOrders or customer_orders . Which should you use? You probably want to use whatever standard is already in place, but if you’re creating a new database, I recommend using_undercores for accessibility. The words “under value” have a different meaning from the word “undervalue”, but the former, with underscores, is always under_value , while the latter is undervalue . With CamelCase, it’s Undervalue versus UnderValue which, since SQL is case-insensitive, are identical. Further, if you have vision problems and are constantly playing around with fonts and sizes to distinguish words, the underscores are much easier to read.  As a side problem, CamelCase is anecdotally harder to read for people for whom English isn’t their first language.  That being said, this is a personal preference and not a strong recommendation.  Plural or Singular Tables?  There’s long been a huge debate amongst experts in database theory about whether or not database tables should be singular ( customer ) or plural ( customers ). Without going into the theory, let me cut the Gordian Knot with a healthy dose of pragmatism: plural table names are less likely to conflict with reserved keywords.  Do you have users? SQL has a user reserved word. Do you want a table of constraints? constraint is a reserved word. Is audit a reserved word but you want an audit table? By simply using the plural form of nouns, most reserved words won’t cause you grief when you’re writing SQL. Even PostgreSQL, which has an excellent SQL parser, has been tripped up when encountering a user table.  Just use plural names and you’re far less likely to have a collision.  Don’t name the ID column “id”  This is a sin I’ve been guilty of for years. When working with a client in Paris, I had a DBA complain when I named my id columns id and I thought he was being pedantic. After all, the customers.id column is unambiguous, but customers.customer_id is repeating information.  And later I had to debug the following:  SELECT thread.* FROM email thread JOIN email selected ON selected.id = thread.id JOIN character recipient ON recipient.id = thread.recipient_id JOIN station_area sa ON sa.id = recipient.id JOIN station st ON st.id = sa.id JOIN star origin ON origin.id = thread.id JOIN star destination ON destination.id = st.id LEFT JOIN route ON ( route.from_id = origin.id AND route.to_id = destination.id ) WHERE selected.id = ? AND ( thread.sender_id = ? OR ( thread.recipient_id = ? AND ( origin.id = destination.id OR ( route.distance IS NOT NULL AND now() &gt;= thread.datesent + ( route.distance * interval '30 seconds' ) )))) ORDER BY datesent ASC, thread.parent_id ASC  Do you see the problem? If the SQL had used full id names, such as email_id , star_id , and station_id , the bugs would have stood out like a sore thumb while I was typing out this SQL , not later when I was trying to figure out both what I did wrong and why I don’t drink as much as I should.  And by request of a few people who couldn’t see the errors, here’s the SQL after it’s corrected. It’s very clear that “star_id” and “email_id”, or “station_id” and “station_area_id” are probably not valid comparisons. As previously mentioned, if SQL had a decent type system, this SQL would not even have compiled.  SELECT thread.* FROM email thread JOIN email selected ON selected.email_id = thread.email_id JOIN character recipient ON recipient.character_id = thread.recipient_id -- station_area_id = character_id is probably wrong JOIN station_area sa ON sa.station_area_id = recipient.character_id -- station_id = station_area_id is probably wrong JOIN station st ON st.station_id = sa.station_area_id -- star_id = email_id is probably wrong JOIN star origin ON origin.star_id = thread.email_id JOIN star destination ON destination.star_id = st.star_id LEFT JOIN route ON ( route.from_id = origin.star_id AND route.to_id = destination.star_id ) WHERE selected.email_id = ? AND ( thread.sender_id = ? OR ( thread.recipient_id = ? AND ( origin.star_id = destination.star_id OR ( route.distance IS NOT NULL AND now() &gt;= thread.datesent + ( route.distance * interval '30 seconds' ) )))) ORDER BY datesent ASC, thread.parent_id ASC  Do yourself a favor and use full names for IDs. You can thank me later.  Column Naming  As much as possible, name columns very descriptively. For example, a temperature column doesn’t make sense for this:  SELECT name, 'too cold' FROM areas WHERE temperature &lt; 32;  I live in France and for anyone here, 32 would be “too hot”. Instead, name that column fahrenheit .  SELECT name, 'too cold' FROM areas WHERE fahrenheit &lt; 32;  Now it’s completely unambiguous.  Also, when you have foreign key constraints, you should name the columns on each side of the constraint identically, if possible. For example, consider this perfectly reasonable, sane, SQL.  SELECT * FROM some_table s JOIN some_other_table o ON o.owner = s.person_id;  That looks sane. There’s really nothing wrong with it. But when you consult the table definition, you discover that some_other_table.owner has a foreign key constraint against companies.company_id . That SQL is, in fact, wrong. Had you used identical names:  SELECT * FROM some_table s JOIN some_other_table o ON o.company_id = s.person_id;  Now it’s immediately clear that we have a bug and you can see it on a single line of code and don’t have to go consult the table definition.  However, it should be noted that this isn’t always possible. If you have a table with a source warehouse and a destination warehouse, you might want a source_id and a destination_id to compare with your warehouse_id . Naming them source_warehouse_id and destination_warehouse_id will make this easier to follow.  It should also be noted that in the example above, owner is more descriptive of the intent than company_id . If you feel this is likely to cause confusion, you can name the column owning_company_id . That can still embed the intent of the column in the name while giving you a strong hint as to its intent.  Avoid NULL Values  Saving the best (or is it worst?) for last! This is a tip that many experienced database developers are aware of, but sadly, it doesn’t get repeated often enough: don’t allow NULL values in your database without an excellent reason.  This will take a bit of time because this is an important, but somewhat complicated topic. First, we’ll discuss the theory, then we’ll discuss their impact on database design, and we’ll finish up with a practical example of the serious problems NULL values cause.  Database Types  In the database, we have various data types , such as INTEGER , JSON , DATETIME , and so on. A type is associated with a column and any value added should conform to the type associated with that column.    But what’s a type? A type is a name, a set of allowed values, and a set of allowed operations. This helps us avoid unwanted behavior. For example, in Java, what happens if you try to compare a string and an integer?  CustomerAccount.java:5: error: bad operand types for binary operator '&gt;' if ( current &gt; threshhold ) { ^ first type: String second type: int  Even if you can’t see by glancing at the code that current &gt; threshhold is comparing incompatible types, the compiler will trap this for you.  Ironically, databases, which store your data—and are your last line of defense against data corruption—are terrible at types! I mean, really, really bad at them. For example, if your customers table has an integer surrogate key, you can do this:  SELECT name, birthdate FROM customers WHERE customer_id &gt; weight;  That, of course, doesn’t make a lick of sense and in a sane world would be a compile-time error.   Many programming languages make it trivial to trap type errors like this but databases make it hard.  But that’s not how databases generally behave, quite possibly because when the first SQL standard was released in 1992  , computers were slow beasts and anything that complicated the implementation would undoubtedly have made databases slow.  And here’s where the NULL value comes into all of this. There is one place where the SQL standard got this right and that’s with the IS NULL and IS NOT NULL predicates. Since the NULL value is, by definition, unknown, you can’t possibly have operators defined for it. That’s why IS NULL and IS NOT NULL exist instead of = NULL and != NULL . And any NULL comparison results in a new NULL value.  If that sounds strange, it becomes much easier if you say “unknown” instead of NULL :   NULL Unknown comparisons result in NULL unknown values.   Ah, now it makes perfect sense!  What does a NULL value mean?  Now that we have the tiniest amount of type theory under our belt, let’s examine the practical implications.  You need to pay a $500 bonus to all employees who earn more than $50K a year in salary. So you write the following SQL.  SELECT employee_number, name FROM employees WHERE salary &gt; 50000;  And you just got fired because your boss earns more than $50K but their salary isn’t in the database (their employees.salary column is NULL ) and the comparison operator can’t compare a NULL with 50000.  And just why is that value NULL ? Maybe their salary is confidential. Maybe the information hasn’t arrived yet. Maybe they’re a consultant and they’re not paid a salary. Maybe they’re paid hourly and are not salaried. There are plenty of reasons why that data might not be available.  The existence or non-existence of data in a column suggests that it depends on something other than just the primary key and your database is possibly denormalized. Thus, columns which might have NULL values are good candidates for creating new tables. In this case, you might have tables for salaries , hourly_rates , none_of_your_business and so on. You’ll still get fired for blindly joining on salaries and missing that your boss doesn’t have one, but at least your database is starting to present you with enough information to suggest that there’s more to the problem than just a salary.  And yes, this was a silly example, but it leads to the final nail in the coffin.  NULLs lead to logical impossibilities  You might have some sense that I’m being a bit pedantic about NULL values, but we have one final example and it’s caused much real-world grief.  Years ago I was in London working for a domain registrar and trying to figure out why a somewhat messy 80 line SQL query was returning incorrect data. There was a particular case where data absolutely should have been returned, but wasn’t. I’m embarassed to say that it took me about a day to track it down and it was a combination of a few things:   I had used an OUTER JOIN  Those can easily generate NULL values  NULL values can cause your SQL to give you incorrect answers   That last statement is something most database developers are unaware of, so let’s look at an example Database In Depth  by C.J. Date. First, a trivial schema with two tables.   suppliers    supplier_id city  s1 London    parts    part_id city  p1 NULL     Those should be pretty clear and it’s harder to get a simpler example.  The following, of course, returns p1 .  SELECT part_id FROM parts;  But what about the following?  SELECT part_id FROM parts WHERE city = city;  That returns no rows since you cannot compare a NULL value with anything—not even another NULL and not even the same  NULL . That seems odd because the city for every given row must the be the same city, even if we don’t know it, right? And that leads us to the following. What does this return? Try to work out the answer before reading it below.  SELECT s.supplier_id, p.part_id FROM suppliers s, parts p WHERE p.city &lt;&gt; s.city OR p.city &lt;&gt; 'Paris';  We get no rows because we can’t compare a NULL city ( p.city ) and thus neither branch of the WHERE clause can evaluate to true.  However, we know that the unknown city either is Paris or it is not Paris . If it’s Paris, the first condition is true ( &lt;&gt; 'London' ). If it’s not Paris, the second condition is true ( &lt;&gt; 'Paris' ). Thus, the WHERE clause must be true, but it’s not, leading to SQL which generates logically impossible results.  That was the bug which bit me in London. Any time you write SQL which can generate or include NULL values you run the risk of having SQL lie to you. It doesn’t happen often, but when it does, it’s devilishly hard to track down.  Summary   Use underscore_names instead of CamelCaseNames  Table names should be plural  Spell out id fields ( item_id instead of id )  Don’t use ambiguous column names  When possible, name foreign key columns the same as the columns they refer to  Add NOT NULL to all column definitions, when possible  Avoid writing SQL which can generate NULL values, when possible   While not perfect, the above database design guidelines will make your database world a better place.  ","title":"Database Design Standards","url":"\/articles\/database-design-standards.html"},{"body":"   The Constraints  Getting Started  The Tyranny of ORMs  Caching  SOAP  Tax Server  The Big Test  What We Delivered  Overview  New Caching Architecture  Data Storage Performance  Implement Staged Processing  Process Streamlining  Repeatable  Caveat     The Devil went down to Georgia. He was lookin' for a soul to steal. He was in a bind 'cause he was way behind. He was willing to make a deal. \"Devil Went Down To Georgia\"—The Charlie Daniels Band   Our client had just won a nice contract but were in a bind. Their legacy codebase, while powerful, was slow. They could not process more than 39 credit card transactions per second. They needed to get to 500 transactions per second for an event lasting 30 minutes. Because the event was highly publicized, there was a tight deadline. They had two weeks to get a proof of concept running, improving their performance by an order of magnitude.  They turned to our company, All Around the World  , because we have a proven track record with them.  We had a senior systems architect, Shawn, and a senior software architect, Noel, on the project. Our managing director, Leïla, oversaw the project and ensured that if we had questions, she had answers. I was brought in from another project because there was simply too much work to do in two weeks. Fortunately, though I didn't know the project, Shawn and Noel knew the system well and helped get me up to speed.  The Constraints  There were several key constraints we had to consider. Our overriding consideration was ensuring that PCI-compliance (Payment Card Industry compliance) was strictly adhered to to ensure that customer data was always protected. Second, the client had developed an in-house ORM (object-relational mapper) many years ago and like all projects, it grew tremendously. While it was powerful, it was extremely slow and had a lot of business logic embedded in it. Third, because we only had two weeks for the first pass, we were given permission to take \"shortcuts\", where necessary, with the understanding that all work was to be thoroughly documented and tested, and easy to either merge, remove, or disable, as needed.  Finally, we could change anything we wanted so long as we didn't change the API or cause any breaking changes anywhere else in the code. There was no time to give customers a \"heads up\" that they would need to make changes.  Getting Started    Source    Because the event only lasted 30 minutes, whatever solution we implemented didn't have to stay up for long. This also meant that whatever solution we implemented had to be disabled quickly if needed. We also knew that only a few customers would use this new \"fast\" solution, and only one payment provider needed to be supported.  Noel immediately started tracing the full code path through the system, taking copious notes about any behavior we would need to be aware of. Shawn was investigating the databases, the servers, the network architecture, and assessing what additional resources could be brought online and tested in two weeks.  I, being new to the project, started by taking a full-stack integration test representing one of these transactions and studied it to learn the code and its behavior. In particular, I wanted to better understand the database behavior as this is often one of the most significant bottlenecks.  I was particularly concerned because the client was in the midst of a large migration from Oracle to PostgreSQL, so making changes at the database level was not an option. We needed to know immediately if this was one of the bottlenecks. I wrote code which would dump out a quick summary of database activity for a block of code. It looked sort of like this:    explain dbitrace( name =&gt; 'Full stack transaction request', code =&gt; sub { $object-&gt;make_full_request(\\%request_data) }, save =&gt; '\/tmp\/all_sql.txt', );  The summary output looked similar to this:  { report =&gt; { name =&gt; 'Full stack transaction request', time =&gt; '3.70091 wallclock secs' }, sql =&gt; { delete =&gt; 1, insert =&gt; 7, select =&gt; 137, update =&gt; 32, total =&gt; 177, } }  We had a problem. Even our client was surprised about the amount of database activity for a single \"buy this thing\" request.  None of the database activity was particularly slow, but there was a lot of it. Deadlocks weren't an issue given that the changed data was only for the current request, but the ORM was killing us.  The Tyranny of ORMs    How it feels to work on most in-house ORMs. Source    I love working with a good ORM, but ORMs generally trade execution speed for developer speed. You have to decide which is more important to you. Further, in two decades of working with ORMs, I have only once seen an in-house ORM which was on-par with, or superior to, commercial or open source products. And it was an ORM optimized for reporting, something many ORMs struggle with.  For our client's ORM, every time a request was made it would gather a bunch of metadata, check permissions, make decisions based on whether or not it was using Oracle or PostgreSQL, check to see if the data was cached, and then check to see if the data was in the database. Instantiating every object was very slow, even if there was no data available. And the code was creating&mdash;and throwing away without using&mdash;hundreds of these objects per request.  We considered using a \"pre-flight\" check to see if the data was there before creating the objects, but there was so much business logic embedded in the ORM layer that this was not a practical solution. And we couldn't simply fetch the data directly because, again, the ORM had too much business logic. We had to reduce the calls to the database.  After an exhaustive analysis, we discovered several things.   Some of the calls were for a \"dead\" part of the system that no one really remembered but was clearly unused.  Numerous duplicate calls were being made to unchanging data. We could cache those objects safely.  A number of calls were being made for important data that wasn't relevant to our code path, so we could skip them.   Our first step at addressing the problem was to ensure that everything was wrapped in configuration variables that allowed us to easily turn on or off different code paths for our project. Fortunately, the client had a system that allowed them to update the configuration data without restarting the application servers, so this made our choice much safer.  Once that was in place, our first pass cut our SQL calls roughly in half, tremendously sped up the code, and the tests still passed. But we were nowhere near 500 transaction per second.  Caching  It's said that the three most common bugs in software are cache invalidation and off-by-one errors. And we had caching problems in spades.  We couldn't afford cache misses during the event, so we needed to \"preheat\" the caches by ensuring all information was loaded on server startup. Except that we had a few problems with this.  First, it was slow because there was a lot of data to cache. If we needed to rapidly bring new servers online during the event, this was not an option.  Second, it consumed a lot of memory and we were concerned about memory contention issues. Disk swapping was not an option.  Third, the app caches primarily used per-server memcached instances, but the data was global, not per-server, thus causing a lot of unnecessary duplication. Shawn already knew about this, so one of the first things he did was set up a secure Redis cluster with failover to replace memcached, where appropriate. Since we only had to heat the cache once, bringing up servers was faster, and we significantly reduced per-server memory consumption.    Now, where did I put that data? Source    The application itself also used heavy in-memory caching (implemented as hashes), which we replaced with pre-loaded shared cache entries, thereby lowering memory requirements even further.  As part of this work, we centralized all caching knowledge into a single namespace rather than the ad-hoc \"per module\" implementations. We also created a single configuration file to control all of it, making caching much simpler for our client. This is one of the many features we added that they still use today.  SOAP    Possibly not the most original image I've come up with ... Source    Another serious bottleneck was their SOAP server. Our client made good use of WSDL (Web Services Description Language) to help their customers understand how to create SOAP requests and SOAP was integral to their entire pipeline. The client would receive a SOAP request, parse it, extract the necessary data, process that data, create another SOAP request, pass this to back-end servers, which would repeat the process with the new SOAP request.  SOAP stands for \"Simple Object Access Protocol.\" But SOAP isn't \"objects\", and it's not \"simple.\" It's a huge XML document, with an outer envelope telling you how to parse the message contents. Reading and writing SOAP is slow . Further, our client's SOAP implementation had grown over the years, with each new version of their SOAP interface starting with cutting-and-pasting the last version's code into a new module and modifying that. There were, at the time of the project, 24 versions,   most of which had heavily duplicated code. To optimize their SOAP, we were facing a problem. However, we dug in further and found that only one version would be used for this project, so our client authorized us to skip updating the other SOAP versions.  We tried several approaches to fine-tuning the SOAP, including replacing many AUTOLOAD (dynamically generated) methods with static ones. In Perl, AUTOLOAD methods are optional \"fallback\" methods that are used when Perl cannot find a method of the desired name. However, this means Perl must carefully walk through inheritance hierarchies to ensure the method requested isn't there before falling back to AUTOLOAD . This can add considerable overhead to a request. For this and other reasons, the use of AUTOLOAD is strongly discouraged.  Replacing these methods was extremely difficult work because the AUTOLOAD methods were used heavily, often calling each other, and had grown tremendously over the years. Many of them were extremely dangerous to pick apart due to their very complex logic. We managed to shave some more time with this approach but stopped before we tackled the most complicated ones. There was only so much risk we were willing to take.  Noel, in carefully reading through the SOAP code, also found several code paths that were slow, but didn't apply to our requests. Unfortunately, we could not simply skip them because the SOAP code was tightly coupled with external business logic. Skipping these code paths would invariably break something else in the codebase. What he proposed, with caution, is the creation of a special read-only \"request metadata\" singleton. Different parts of the code, when recognizing they were in \"web context\", could request this metadata and skip non-essential code paths. While singletons are frowned upon by experienced developers, we were under time pressure and in this case, our SOAP code and the code it called could all consult the singleton to coordinate their activity.  The singleton is one of the many areas where deadlines and reality collide. Hence, the \"Devil Went Down to Georgia\" quote at the beginning of this case study. We were not happy with this solution, but sometimes you need to sacrifice \"best practices\" when you're in an emergency situation. We alerted our client to the concern so they could be aware that this was not a long-term solution.  We also leveraged that singleton to allow us to alert the back-end that encrypted transaction data was available directly via Redis. The back-end merely needed to decrypt that data without the need to first deserialize SOAP and then decrypt the data. Another great performance improvement. Sadly, like many others improvements, it was specific to this one customer and this one payment provider.  However, this approach allowed us to remove even more SQL calls, again providing a nice performance boost.  Tax Server    In the 80s, I used to do accounting by hand. Source    By this time, with the above and numerous other fixes in place, we were ready to do our initial QA runs. We didn't expect to hit our 500 TPS target, but we were pretty confident we had achieved some major speed gains. And, in fact, we did have some pretty impressive speed gains, except that periodically some of our requests would simply halt for several seconds. Due to the complicated nature of the system, it wasn't immediately clear what was going on, but we finally tracked down an issue with the tax server.  Our client was processing credit card transactions and for many of them, taxes had to be applied. Calculating taxes is complicated enough that there are companies that provide \"tax calculation as a service\" and our client was using one of them. Though the tax service assured us they could handle the amount of traffic we were sending, they would periodically block. It wasn't clear if this was a limitation of their test servers that we would avoid with their production servers, but we could not take this chance.  We tried several approaches, including caching of tax rates, but the wide diversity of data we relied on to calculate the tax meant a high cache miss rate, not allowing us to solve the problem. Finally, one of our client's developers who knew the tax system fairly well came up with a great solution. He convinced the tax service to provide a \"one-time\" data dump of tax rates that would not be valid long but would be valid long enough for our short event. We could load the data into memory, read it directly, and skip the tax server entirely. Though the resulting code was complicated, it worked, and our requests no longer blocked.  The Big Test  With the above, and many other optimizations, we felt confident in what we delivered and in a late-night test run, our client was ecstatic. We were handling 700 transactions per second, almost twenty times faster than when we started. Our client hired us because they knew we could deliver results like this. Now it was time for our first production run, using beefier servers and a much faster network.  Cue the \"sad trombone\" music. We were only running about 200 to 300 transactions per second. As you may recall, the client had been migrating from Oracle to PostgreSQL. The QA servers were far less powerful than the production servers, but they were running PostgreSQL. The production servers were running Oracle. Oracle is the fast, beastly sports car that will quickly outpace that Honda Accord you're driving ... so long as the sports car has an expert driver, constant maintenance, and an expensive, highly trained pit crew to look after it.    How I felt after our first production test run. Source    Out of the box, PostgreSQL just tends to be fast. Often companies find that a single PostgreSQL DBA working with competent developers is enough to easily handle their traffic. But we had Oracle. And it wasn't going away before the deadline. Our first two weeks was up and our proof of concept worked, but greatly exceeding client requirements still wasn't fast enough.  However, this was good enough for the proof of concept, so it was time to nail this down. It was going to be more nights and weekends for us, but we were having a huge amount of fun.  Oracle wasn't the only problem, as it turns out. This faster, higher capacity network had switches that automatically throttled traffic surges. This was harder to diagnose and work around but alleviated the networking issues. Interestingly, it was Leïla who spotted that problem simply by listening carefully to what was being said in meetings. She has this amazing ability to hear when something isn't quite right.  With the use of Oracle being a bottleneck, we had to figure out a way to remove those final few SQL statements. Our client suggested we might be able to do something unusual and it's based on how credit cards work.    Source    When you check into a hotel and reserve your room with a credit card, your card is not charged for the room. Instead, they \"authorize\" (also known as \"preauthorize\") the charge. That lowers the credit limit on your card and reserves the money for the hotel. They will often preauthorize for over the amount of your bill to allow a small wiggle room for incidental charges, such as food you may have ordered during your stay.  Later, when you check out, they'll charge your card. Or, if you leave and forget to check out, they have a preauthorization, so they can still \"capture\" the preauthorization and get the money deposited to their account. This protects them from fraud or guests who are simply forgetful.  In our case, the client often used preauthorization for payments and after the preauthorization was successful, the card would be charged.  It worked like this: the front-end would receive the request and send it to the back-end. The back-end would submit the request for preauthorization to the payment provide and the payment provider would respond. The back-end sent the response to the front-end and if the preauthorization was denied, the denial would be sent back to the customer. However, usually it was approved and the front-end would then submit the charge back to the back-end.   The back-end would then submit the preauthorized charge to the payment provider and lots of network and database activity helped to keep the server room warm. The entire process was about as fast as a snail crawling through mud.  What the client pointed out is that the money was reserved, so we could charge the card after the event because we had already preauthorized it. Instead, we could serialize the charge request and store it in an encrypted file on the front-end server (with the decryption key unavailable on those servers). Later, a utility we wrote read and batch processed those files after the event, allowing us to finish the transaction processing asynchronously.  This worked very well and by the time we were done, our changes to the code resulted in a single SQL SELECT statement. Further, after this and a few other tweaks, we were easily able to hit 500 transactions per second on their production system. When the event happened shortly after, the code ran flawlessly.  What We Delivered    Source    Overview  Increase the performance of Business Transactions from 39\/sec to 500+\/sec.  The overall implemented solution could not change the outward facing API calls as end users of the service were not to be impacted by back-end changes. The solution could also not effect the traditional processing as clients would be issuing both traditional and streamlined transactions.  New Caching Architecture  Implement a new redundant shared caching architecture using Redis as a data store, to replace the existing on application server memcached instances.  Update existing application code to remove the use of internal application caches (implemented as hashes), and replace with pre-loaded shared cache entries.  The new caching system caches items using both the legacy and the new caching system, so that production systems can be transitioned to the unified caching system without downtime.  Allowed a cache storage back-end to define its storage scope, so that caches can be managed based on the storage locality:   \"process\": cache is stored in local per-process memory  \"server\": cache is stored on the local server, and shared by all processes on the server  \"global\": cache is stored globally, and shared by all processes on any server   Data Storage Performance  Create an additional data storage path that replaces storing intermediate data in the back-end shared database with the shared caching architecture.  Remove the use of a third party provider API call, and replace with a static table lookup.  Implement Staged Processing  As data processing in real time is not a requirement, add a data path that allows for separation of the front end API responses from the third party back-end processing. This allowed for heavy back-end processes to be deferred and processed in smaller batches.  Process Streamlining  Worked closely with the client to determine which parts of the process are required during the processing, and which can be bypassed to save overhead.  Code review and clean up. Refactor code base to remove extraneous or ill-performing legacy code.  Repeatable  The process was re-engineered to be configurable allowing the same pipeline to be reimplemented for future events\/customer requirement through simple documented configuration options, while not impacting traditional processing.  Caveat  This information was compiled from our reports during this project and not written while the project was ongoing. Some events may not have happened in the exact sequence listed above.  ","title":"Case study: 500 TPS","url":"\/articles\/project-500.html"},{"body":"    Introduction  Programming A New Feature  Use Control Panels  Don’t Use Control Panels  Permanent Features  “Sampled” Switches  Naming Conventions  Operations  Errors  Communications  Simplicity  Clean Features Regularly  Don’t Check Features at Compile=Time  Conclusion     Introduction  One of the best things to come out of the modern DevOps movement is the aggressive push for “feature switches” (also known as “feature flags” or “feature toggles”). At All Around the World  , we strongly recommend them for our clients and sometimes we implement them, but I’m still surprised that many companies don’t use them. They’re dead-simple to build and they’re a powerful tool for site reliability. Unfortunately, many articles discuss how to build them or use them, but best practices are neglected. So we’ll skip the implementation and instead focus on using feature switches effectively.  Note: the discussion below refers to “sites” because this is in the context of websites. Obviously, it doesn’t have to be.  A couple of decades ago when I was working for a small web-development firm in Portland, Oregon, we released a new version of our software that had a critical bug in a new feature. The bug was serious enough that it took down the entire web site. We had to revert to an earlier version (from CVS, the good ‘ol days!) and redeploy the web site. Our largest client’s site was down for an hour and that was not good. Had we used feature switches, we could simply have turned the feature off and the client might not have even noticed.  Note that you don’t need to implement everything I suggest in your first pass; you just need to get up and running and comfortable with feature switches. Once you do, you can better understand the additional tooling support needed for your use case.  Programming A New Feature  For the developer, using feature switches is straight-forward. Imagine that you always show a list of categories for products you sell, but you’re fetching it from the database every time. Developers argue for caching the list, but sometimes it changes and arguments over cache invalidation stall development. Finally, an agreement is reached, but you don’t want to roll the new functionality out blindly, so you use a feature switch. Here’s some pseudo-code in Perl.  my $categories; if ( feature_on('cached_categories') ) { $categories = $cache-&gt;category_names; } else { $categories = $app-&gt;category_names; }   Side note: the above is shown for simplicity. In reality, $app variables often imply some kind of global state and if it’s mutable, that can be dangerous, but we’re keeping it simple.   In the above, if the cached_categories feature is on, we fetch features from the cache. If it’s not, we fetch it from the database. There are numerous ways this can be done on the front-end, but we’ll skip those for the purposes of this article.  Why would you bother to use a feature switch for the above? You’ve tested it. The customer won’t see a difference. You know it works. Right?  No, you don’t. What if your cache invalidation doesn’t work quite as expected? What if your cache isn’t loaded? What if your brand-new caching system isn’t as reliable as you thought and you actually need the following?  my $categories; if ( feature_on('cached_categories') ) { $categories = $cache-&gt;category_names || $app-&gt;category_names; } else { $categories = $app-&gt;category_names; }  Sound paranoid? One definition of a good programmer is someone who looks both ways before crossing a one-way street. So you roll out your new code with your feature turned off and after you verify everything is good, you turn on your feature. How?  Use Control Panels  Typically, a good system for feature switches should have a web-based control panel that lists each feature and shows, amongst other things:   The name of the feature  A description of what the feature is for  The current on-off state  Whether the feature is permanent (more on that later)  An “on-off” history of the feature   The “on-off” history is simply an audit log. At minimum, you want:   Who toggled the feature  The date it was turned on\/off  Why it was turned on\/off  Possibly the release-tag of the software   An audit log of the history can be immensely useful if you want to turn on a feature that was turned off for several months and you don’t remember why.   For companies that make heavy use of feature switches, this can be invaluable. If your company is structured well enough, you might even want to include what team the feature belongs to, or what departments are responsible for it. Because so many companies are structured differently, it’s hard to give clear guidance without knowing the specifics.  Don’t Use Control Panels  Yes, feature switches are usually used via control panels and they make it easy to turn features on and off, but there are caveats. It sometimes happens that enabling a new feature causes every page on your site to crash, taking down the control panel! Oops. Having a control panel running on a dedicated server, separate from the servers using the features can help to mitigate this. Having command-line tools to turn features on or off can also be useful. Do not rely on control panels alone . Nothing’s worse than having everything web-based and losing web access.  Permanent Features  One thing often overlooked for features switches is that some of them should be permanent. Imagine that your site goes viral and you’re hitting unprecedented amounts of load and you’re struggling. Some sites handle this by having optional features on the site wrapped in permanent feature switches. If your site is struggling under a heavy load, being able to toggle off optional features such as autocomplete or “related searches” that aren’t critical can help you manage that load. Your site experience degrades gracefully instead of simply crashing.  “Sampled” Switches  Before turning on a switch, it’s often useful to turn it on only for a subset of requests. For example, you might only want 10% of users to experience it. If a feature turns out to be disastrous, this can limit the impact. However, this can be odd for someone who sees a new feature, refreshes a web page, and that feature disappears because they are no longer in the randomly selected subset. Making such features persistent per user can be tricky, though. Do you rely on cookies? Your mobile app may not use cookies. Or someone might switch browsers, such as switch from their work to their personal computer.  Don’t stress about this (unless you’re doing A\/B testing). “Sampled” features are short-lived. As feature switches are primarily a site reliability tool, the limited sample run is just there to avoid disaster and you should go 100% as soon as is feasible.  Naming Conventions  For one of our clients, they have no naming convention for their features, but they use features heavily. We wrote code to scan our CODEOWNERS file to dump a list of all features in our code in order to regularly monitor them. This allowed us to do heavy cleanup when we found many features that had been running successfully for years, but were never cleaned. Even this was limited because many of “our” features were scattered in other parts of the codebase.  To deal with this, establishing a naming convention early on can make it easier to search for features. Are you on the advertising team and you have a new feature offering a discount for first-time buyers? You use ADV- as a prefix: ADV-FIRST_TIME_BUYER . Searching for all features prefixed with ADV- gives you an overview of the features your team has running.  When you identify a naming convention, do not put ticket numbers in the feature name. Many companies switch ticketing systems and old ticket numbers become useless. Instead, having a “Ticket URL” field on your form when you create a feature allows others to click through and understand the intent of that feature.  Operations  Errors  If you dump out rich error data to error logs, include a list of running features, sorted by last start time. Frequently, new errors show up due to recently introduced features and this makes it much easier to debug, especially if first time buyers sometimes can’t place orders and you see the ADV-FIRST_TIME_BUYER feature was just switched on. This makes life much easier.  Communications  Let other teams know when you’re about to start a new features, especially your site reliability engineers (or equivalent). Give them a heads-up about the intent of the feature and what to look for if their are issues. Such communication can allow tracking down issues faster.  Simplicity  Feature switches are a site reliability tool. Keep them simple. You don’t want your site to break because someone made the feature switch tool too complicated. This is one of the reasons I prefer features to be “booleans” (in other words, “on or off”). Some recommend feature tools that allow you to do this:  my $categories; if ( 'enhanced' eq feature_on('SOME_FEATURE') ) { $categories = $app-&gt;categories_enhanced; } elsif ( 'expanded' eq feature_on('SOME_FEATURE') ) { $categories = $app-&gt;categories_expanded; } elsif ( feature_on('SOME_FEATURE') ) { warn &quot;Unknown feature category for SOME_FEATURE&quot;; } $categories \/\/= $app-&gt;category_names;  Does the above start to look complicated? Well, kinda. If you allow more than just “on” or “off”, you need to check for the case where the feature value doesn’t match expectations. Maybe someone added a new value to the feature? Maybe expanded was actually expandable and you misspelled it in your code. By allowing these multiple values, you’re complicating code that’s a site reliability tool. KISS: Keep it simple, stupid. Feature switches are there to make life easier. Don’t overthink them.  Clean Features Regularly  Aside from “permanent” features that you want to always be able to switch off, it’s important to regularly clean features. Otherwise, you get into messes like this:  if (feature_on('FEATURE_1')) { # do something if (feature_on('FEATURE_2')) { # do something else } elsif (feature_on('FEATURE_3') { # do yet another thing if (feature_on('FEATURE_4')) { ... } } }  Not only does that make the code harder to read, you can wind up with cases where features work well on their own, but there could be some unknown interaction between conflicting features that won’t be noticed unless you have the right combination of features toggled on or off. This is miserable to debug.  Don’t Check Features at Compile=Time  This one is a subtle trap. Feature switches are designed to allow you to quickly turn a feature on or off while the system is still running. However, if your code that checks the feature does so too early and not at runtime, you may find that you can’t turn your feature on or off! Here’s an example in Perl:  package Some::Package; sub import ($class) { if ( feature_on('MY_COOL_FEATURE') ) { ... } } # elsewhere package Another::Package; use Some::Package; ...  In the above, the use Some::Package statement will cause Some::Package to call import and that will check the feature switch, but that will happen only once. Toggling the feature switch later will likely not affect the code while Another::Package is resident in memory.  Worse, if some other code is loaded later , but the feature switch has been toggled, you may find that different parts of your codebase do not agree on whether or not a feature is running. This is not a good place to be.  Conclusion  Feature switches are an incredibly powerful site reliability tool which can often eliminate the need to redeploy your software if something goes wrong. This can improve your customer experience and dramatically cut costs. However, like any new system you implement, you want to take some time up front deciding how it will be used and what your best practices are.  You will probably find that the best practices you develop for your company differ from what I describe above. That’s OK. I’m not dogmatic. But planning for this in advance can give you greater confidence in pushing new code that might otherwise be cause for concern.  If you’d like top-notch experienced developers with decades of experience to come in and make your life easier, let me know . Our work at All Around the World  , is focused on pragmatic development custom-tailored to client needs.  ","title":"Feature Switch Best Practices","url":"\/articles\/feature-switch-best-practices.html"},{"body":"   Full disclosure: I just found out that because of my contributions to significant open source projects, I get access to Github Copilot for free. That being said, even before I realized that, I was using the free trial and I was impressed.   A few years ago, a client contracted with us to build out a system where they’d be automating many common tasks their people did. However, it was a political minefield. The employees thought management was trying to automate them out of jobs, so they were not always cooperative. Management insisted they were trying to free up employee time for more complex tasks, but the employees were suspicious.  Today’s AI revolution is having similar effects. People are losing their jobs to ChatGPT. Companies are turning to Stable Diffusion and other AI systems to generate images. And now Github Copilot  is here to write code for us.  I tend to be a late adopter of software. It’s not that I’m a Luddite. I just prefer to focus on building things rather than learning the hot new technology du jour. But my explorations into ChatGPT taught me pretty quickly that if you’re willing to accept the quirks, AI can be an amazing tool. So I decided to give Copilot a try.  I use vim. If you have a different setup, you’ll need to hit your favorite search engine for instructions.  First, you’ll need neovim or a relatively recent vim and install the Github Copilot vim plugin  . I confess that after two decades with vim, and having a highly customized vim setup, I was a bit nervous about whether or not Copilot was going to play well with my setup. There were a few slight bumps in the road.  First, I wanted to make sure Copilot was only active in certain filetypes. I added the following to the bottom of my ~\/.vimrc :  let g:copilot_filetypes = { \\ 'gitcommit': v:true, \\ 'markdown': v:true, \\ 'yaml': v:true, \\ 'perl': v:true \\ }  That gives me Copilot in git commit messages, markdown files, YAML files, and Perl.  If you have extremely large files, you may want to disable Copilot in those  :  autocmd BufReadPre * \\ let f=getfsize(expand(&quot;&lt;afile&gt;&quot;)) \\ | if f &gt; 100000 || f == -2 \\ | let b:copilot_enabled = v:false \\ | endif  I was pretty sure that my auto-completion hack wouldn’t work, so I disabled that first:  &quot; function! CleverTab() &quot; I need to be cleverer &quot; if strpart( getline('.'), 0, col('.')-1 ) =~ '^\\s*$' &quot; return &quot;\\&lt;Tab&gt;&quot; &quot; else &quot; return &quot;\\&lt;C-N&gt;&quot; &quot; endfunction &quot; inoremap &lt;Tab&gt; &lt;C-R&gt;=CleverTab()&lt;CR&gt;  Yeah, I should use a plugin for that, but I’m lazy. It worked for me.  Then I started playing around with Copilot, but for some reason, sometimes it would show me the code it was suggesting, but when I hit “tab”, it wouldn’t insert it. It was frustrating, until I remembered I had this in my ~\/.vimrc :  noremap &lt;leader&gt;pp :set invpaste&lt;cr&gt;  I’m often cutting-n-drooling things into my editor, so I hit ,pp to toggle paste mode. When I realized that, I knew I just needed to hit ,pp again to let Copilot work its magic.  I’ve seen quite a few people talk about how Copilot is going to put programmers out of work, but I was pretty sure that while it could write simple code, it wasn’t going to be able to write code for complex business logic. I was sort of wrong. Copilot quickly picked up my style and if I named things intelligently, it often offered perfect suggestions. In fact, a few times it was offering suggestions that were better than what I thought of at the time. Humbling.  Of course, like ChatGPT and friends, Copilot is prone to hallucinations. It’s tried to write documentation for methods that don’t exist. It’s tried to write code calling methods that don’t exist. Sometimes it writes code that works, but has nothing to do with what I need. However, so long as I’m vigilant, I can catch those errors. And sometimes, it’s just a matter of hitting “tab” a few times to get the code I want.  This morning, I was working on an immutable ORM I’m building for a client. At one point, Copilot added the following comment to my code:  METHOD: foreach my $method (keys %methods) { my $new_method = $self-&gt;metadata('columns')-&gt;{$method}; if ( $method eq $new_method ) { # don't install methods if they already exist next METHOD; ...  I chuckled at comment. It was code being used to auto-generate methods and of course I wanted to install those methods.  And then I read the comment again. That next METHOD; meant I’d be skipping that method. I had a bug in my code and Copilot caught it for me. It was a silly, obvious bug. I was knee-deep in a huge change that impacted 27 files and tiny bugs like this are easy to miss. Thank goodness for tests, but doubly-thank goodness for Copilot.  Did I say “humbling” already? Yeah, humbling.  And then what blew me away happened this weekend. I was porting the OpenSky API  from Python to Perl. What I often do when porting code is to take the code and paste it into my editor as a comment and follow along while porting. I was using the following Python:  def get_flights_from_interval(self, begin, end): &quot;&quot;&quot; Retrieves data of flights for certain time interval [begin, end]. :param int begin: Start of time interval to retrieve flights for as Unix time (seconds since epoch). :param int end: End of time interval to retrieve flights for as Unix time (seconds since epoch). :return: list of FlightData objects if request was successful, None otherwise. :rtype: FlightData | None &quot;&quot;&quot; if begin &gt;= end: raise ValueError(&quot;The end parameter must be greater than begin.&quot;) if end - begin &gt; 7200: raise ValueError(&quot;The time interval must be smaller than 2 hours.&quot;) params = {&quot;begin&quot;: begin, &quot;end&quot;: end} states_json = self._get_json( &quot;\/flights\/all&quot;, self.get_flights_from_interval, params=params ) if states_json is not None: return [FlightData(list(entry.values())) for entry in states_json] return None  As soon as I typed sub get_flights_from_interval , Copilot suggested this:  sub get_flights_from_interval ( $self, $begin, $end ) { if ( $begin &gt;= $end ) { croak 'The end time must be greater than or equal to the start time.'; } if ( ( $end - $begin ) &gt; 7200 ) { croak 'The time interval must be smaller than two hours.'; } my %params = ( begin =&gt; $begin, end =&gt; $end ); my $route = '\/flights\/all'; my $response = $self-&gt;_get_response( route =&gt; $route, params =&gt; \\%params ) \/\/ []; if ( $self-&gt;raw ) { return $response; } return OpenSky::API::Flights-&gt;new($response); }  The only change I made to it was rewriting the error messages.  There are a two interesting this about this. One, it worked perfectly the first time. Two, notice that $self-&gt;raw ? That’s something I added to other methods to return the raw data structures instead of objects. Copilot knew to add it there.  Also, the Python code returns None if it doesn’t find anything, meaning that you always have to remember to guard your code with an if check. My code always returns an OpenSky::API::Flights object, but with an empty iterator if there are no results. If you forget an if check, the consuming code will still work. If you need to check if you have results, you can still call $flights-&gt;count to see if anything is there. Copilot knew to add that, too.  Copilot is not only suggesting code, but accommodating my style.  To be fair, there are serious concerns about Copilot. It’s trained on code from a variety of sources, including code that is copyrighted. There’s an ongoing class action lawsuit as a result of this  .  But rather go into a number of issues with Copilot, I’ll point to the elephant in the room: developers are going to lose jobs because of this. This technology is in its infancy and it’s already writing code that’s better than many developers I’ve met. As the technology matures, more and more developers will find that simply writing good prompts will take care of much of their work for them. Already I’m finding that Copilot is increasing my productivity, but at the same time, I need to be on guard for subtle bugs that it might introduce. Since I’m not writing the code, I risk not thinking about it as much as I should. Fortunately, I’m fanatical about testing, so this limits the risk.  But as more companies shift to tools like this, they’ll be looking at senior developers like myself, developers with decades of experience, and they’ll be asking themselves why they’re paying us so much when Copilot can do the work for them. Junior devs might be finding it easier to get work since they don’t charge as much money and I think it’s currently unknown if the quality of our software will improve as a result.  I don’t know what the future holds, but one part is clear: we’re getting closer to being able to describe our software and having it written for us.  Note: the OpenSky code I wrote using Copilot is available on github  and the CPAN  . I’ve already had one bug report (my bug, not Copilot’s) and I fixed it and released this morning, along with more tests.  ","title":"Using Github Copilot with Vim","url":"\/articles\/using-github-copilot-with-vim.html"},{"body":"  You may recall that one Jack Sweeney created ElonJet  , a service to track Elon Musk’s jet in real time. This led to his account being suspended by Twitter, but you can find him on Mastodon  .  He has a public repository for tracking planes  , but it’s written in Python. I wanted to do the same for Perl, so I did. I wrote WebService::OpenSky  (also available on GitHub  ), which is a Perl wrapper around the OpenSky Network API  . From OpenSky Network’s home page  :   The OpenSky Network is a non-profit community-based receiver network which has been continuously collecting air traffic surveillance data since 2013. Unlike other networks, OpenSky keeps the complete unfiltered raw data and makes it accessible to academic and institutional researchers. With over 30 trillion ADS-B, Mode S, TCAS and FLARM messages collected from more than 5000 sensors around the world, the OpenSky Network exhibits the largest air traffic surveillance dataset of its kind. The mission of our non-profit association is to support open global air traffic research by universities and other not-for-profit institutions. You can find more information on our blog  .   I won’t go into great detail about the API, but it’s pretty simple. You can read the docs to understand more, but here’s a quick example:  use WebService::OpenSky; my $musks_jet = 'a835af'; my $openapi = WebService::OpenSky-&gt;new; my $days = shift @ARGV || 7; my $now = time; my $then = $now - 86400 * $days; my $flight_data = $openapi -&gt;get_flights_by_aircraft( $musks_jet, $then, $now ); say &quot;Jet $musks_jet has &quot; . $flight_data-&gt;count . &quot; flights&quot;;  As of this writing, that prints out: Jet a835af has 3 flights  The $flight_data object allows you to iterate over the flights, but it’s pretty limited. I wanted more. I wanted something like this:  Flight #1. Departed Monday, 29 May, 10:42 AM UTC from Austin-Bergstrom International Airport, Austin, Texas, US Flight #1. Arrived Monday, 29 May, 04:34 PM UTC at Ted Stevens Anchorage International Airport, Anchorage, Alaska, US Flight #2. Departed Thursday, 01 Jun, 04:32 AM UTC from unknown airport, unknown city Flight #2. Arrived Thursday, 01 Jun, 04:45 PM UTC at Austin-Bergstrom International Airport, Austin, Texas, US Flight #3. Departed Thursday, 01 Jun, 04:32 AM UTC from unknown airport, unknown city Flight #3. Arrived Thursday, 01 Jun, 04:45 PM UTC at unknown airport, unknown city  As you can see, the data itself is spotty, but the older the data, the more accurate it appears to be. So how did we get there?  First, we need to get the ICAO 24-bit hexadecimal code  for Musk’s private jet. It turns out he has three of them, but I’m only tracking the one he’s using the most. A quick search for “icao 24 Elon Musk’s jet” revealed this Hacker News discussion of it  . Various other sources confirmed this, with the N628TS registrations having the icao number of a835af .  Next, we need to get the flight data. The OpenSky API allows you to get flight data and that’s shown in the script above. However, it’s not very useful. I wanted to see the actual airport names and times (the times returned by the API are Unix epochs  ).  Getting the ICAO codes for the airports was a bit harder. Since I’m doing this open source, I eventually tracked down a fairly active GitHub repository with airport data  . I wanted to use the Geo::ICAO  module to convert the ICAO data, but it’s out-of-date and missing a lot of airports. Thus, I wrote this code:  sub get_icao_data { my $url = 'https:\/\/raw.githubusercontent.com\/mborsetti\/airportsdata\/main\/airportsdata\/airports.csv'; my $ua = Mojo::UserAgent-&gt;new; return eval { my $response = $ua-&gt;get($url)-&gt;res; my %lookup; if ( $response-&gt;is_success ) { my ( $fh, $filename ) = tempfile(); print {$fh} $response-&gt;body; close $fh; my $data = csv( in =&gt; $filename, headers =&gt; &quot;auto&quot;, ); %lookup = map { $_-&gt;{icao} =&gt; $_ } @$data; } \\%lookup; }; }  If any airports are missing from that (and it appears there may be some issues), I fall back to Geo::ICAO . The solution has been working well so far. Obviously, paid services are more accurate, but this is good enough for my needs.  I’ll skip explaining the rest of the code, but here’s a link to the full script  . Running that as perl lights.pl --days 30 outputs:  Flight #1. Departed Sunday, 07 May, 08:39 PM UTC from Miami-Opa Locka Exec Airport, Miami, Florida, US Flight #1. Arrived Sunday, 07 May, 11:01 PM UTC at Austin-Bergstrom International Airport, Austin, Texas, US Flight #2. Departed Monday, 08 May, 06:40 PM UTC from Austin-Bergstrom International Airport, Austin, Texas, US Flight #2. Arrived Monday, 08 May, 07:13 PM UTC at Corpus Christi International Airport, Corpus Christi, Texas, US Flight #3. Departed Monday, 08 May, 07:47 PM UTC from Corpus Christi International Airport, Corpus Christi, Texas, US Flight #3. Arrived Monday, 08 May, 08:21 PM UTC at Austin-Bergstrom International Airport, Austin, Texas, US Flight #4. Departed Monday, 08 May, 08:59 PM UTC from Austin-Bergstrom International Airport, Austin, Texas, US Flight #4. Arrived Monday, 08 May, 09:29 PM UTC at Corpus Christi International Airport, Corpus Christi, Texas, US Flight #5. Departed Monday, 08 May, 09:55 PM UTC from Corpus Christi International Airport, Corpus Christi, Texas, US Flight #5. Arrived Monday, 08 May, 10:10 PM UTC at Farm Services Inc Airport, Rio Hondo, Texas, US Flight #6. Departed Tuesday, 09 May, 02:07 PM UTC from unknown airport, unknown city Flight #6. Arrived Tuesday, 09 May, 05:19 PM UTC at Napa County Airport, Napa, California, US Flight #7. Departed Tuesday, 09 May, 06:37 PM UTC from Napa County Airport, Napa, California, US Flight #7. Arrived Tuesday, 09 May, 07:06 PM UTC at Norman Y Mineta San Jose International Airport, San Jose, California, US Flight #8. Departed Friday, 12 May, 06:25 AM UTC from Norman Y Mineta San Jose International Airport, San Jose, California, US Flight #8. Arrived Friday, 12 May, 07:15 AM UTC at Jack Northrop Field\/Hawthorne Municipal Airport, Hawthorne, California, US Flight #9. Departed Saturday, 13 May, 04:18 AM UTC from Jack Northrop Field\/Hawthorne Municipal Airport, Hawthorne, California, US Flight #9. Arrived Saturday, 13 May, 05:27 AM UTC at unknown airport, unknown city Flight #10. Departed Sunday, 14 May, 10:27 PM UTC from unknown airport, unknown city Flight #10. Arrived Monday, 15 May, 07:38 AM UTC at Paris-Le Bourget Airport, Paris, Ile-de-France, FR Flight #11. Departed Monday, 15 May, 06:19 PM UTC from Paris-Le Bourget Airport, Paris, Ile-de-France, FR Flight #11. Arrived Tuesday, 16 May, 03:48 AM UTC at Seidel Ranch Airport, Elroy, Texas, US Flight #12. Departed Friday, 19 May, 01:10 PM UTC from Austin-Bergstrom International Airport, Austin, Texas, US Flight #12. Arrived Friday, 19 May, 03:59 PM UTC at Norman Y Mineta San Jose International Airport, San Jose, California, US Flight #13. Departed Saturday, 20 May, 06:35 AM UTC from Norman Y Mineta San Jose International Airport, San Jose, California, US Flight #13. Arrived Saturday, 20 May, 11:12 AM UTC at Rhode Island Tf Green International Airport, Providence, Rhode Island, US Flight #14. Departed Saturday, 20 May, 07:54 PM UTC from unknown airport, unknown city Flight #14. Arrived Saturday, 20 May, 11:17 PM UTC at Seidel Ranch Airport, Elroy, Texas, US Flight #15. Departed Sunday, 21 May, 12:12 AM UTC from Austin-Bergstrom International Airport, Austin, Texas, US Flight #15. Arrived Sunday, 21 May, 02:45 AM UTC at Los Angeles International Airport, Los Angeles, California, US Flight #16. Departed Sunday, 21 May, 06:03 PM UTC from Los Angeles International Airport, Los Angeles, California, US Flight #16. Arrived Sunday, 21 May, 08:32 PM UTC at Seidel Ranch Airport, Elroy, Texas, US Flight #17. Departed Tuesday, 23 May, 02:41 PM UTC from Austin-Bergstrom International Airport, Austin, Texas, US Flight #17. Arrived Tuesday, 23 May, 05:38 PM UTC at Norman Y Mineta San Jose International Airport, San Jose, California, US Flight #18. Departed Thursday, 25 May, 03:40 PM UTC from Norman Y Mineta San Jose International Airport, San Jose, California, US Flight #18. Arrived Thursday, 25 May, 04:27 PM UTC at Jack Northrop Field\/Hawthorne Municipal Airport, Hawthorne, California, US Flight #19. Departed Friday, 26 May, 01:08 AM UTC from Jack Northrop Field\/Hawthorne Municipal Airport, Hawthorne, California, US Flight #19. Arrived Friday, 26 May, 01:55 AM UTC at Norman Y Mineta San Jose International Airport, San Jose, California, US Flight #20. Departed Friday, 26 May, 07:00 AM UTC from Norman Y Mineta San Jose International Airport, San Jose, California, US Flight #20. Arrived Friday, 26 May, 09:37 AM UTC at Seidel Ranch Airport, Elroy, Texas, US Flight #21. Departed Monday, 29 May, 10:42 AM UTC from Austin-Bergstrom International Airport, Austin, Texas, US Flight #21. Arrived Monday, 29 May, 04:34 PM UTC at Ted Stevens Anchorage International Airport, Anchorage, Alaska, US Flight #22. Departed Thursday, 01 Jun, 04:32 AM UTC from unknown airport, unknown city Flight #22. Arrived Thursday, 01 Jun, 04:45 PM UTC at Austin-Bergstrom International Airport, Austin, Texas, US Flight #23. Departed Thursday, 01 Jun, 04:32 AM UTC from unknown airport, unknown city Flight #23. Arrived Thursday, 01 Jun, 04:45 PM UTC at unknown airport, unknown city  As you can see, he arrived in Paris on Monday, 15 May, 2023 at 07:38 AM UTC. He spent a few hours there before flying back to the Seidel Ranch airport in Texas (that was the first airport which let me know that Geo::ICAO data was incomplete). A quick search revealed he was in Paris to meet with French president Emmanuel Macron.  Have fun!  ","title":"Tracking Elon Musk's Plane with Perl","url":"\/articles\/tracking-elon-musks-plane-with-perl.html"},{"body":"  The more I dig into the logic of Perl, the more I find that it uses constructive (intuitionist) instead of classical logic. But Perl developers think in terms of the latter, not the former. This causes us to often reason incorrectly about our software.     For example, if we have this:  if ( $x &lt; 3 ) { ... } else { ... }  We tend to think that $x is a number and $x &gt;= 3 holds in the else block. That might be true in a strongly typed (uses classical logic) language, but that’s not necessarily true in either Perl or constructive logic.  Here’s a counter-example in Perl:  my $x = {}; # a reference to a hash if ($x &lt; 3) { say &quot;if&quot;; } else { say &quot;else&quot;; }  That will print else because my $x = {}; say $x + 0; will often print a number like 4966099583. That’s the address of $x .  You won’t even get a warning.  Not useful.  So the else branch in Perl simply says the previous case didn’t hold, but it says nothing about the actual value of the number.  We can assert that $x is an integer and die if it’s not, but many don’t bother with that.  Classical versus Constructive  In classical logic, there’s the law of the excluded middle. Something is either true or it’s not. In constructive logic, things are true, not true, or other (unknown).  If $x is an integer, we know that it’s either 3 or it’s not, but we might not know it yet .   Note: Constructive (intuitionist) logic isn’t a mere mathematical curiosity. The well-known physicist Nicolas Gisin has written a short, easy-to-read paper showing how intuitionist logic might prove that there really is an “arrow of time” in physics  .   I find the idea of constructive logic useful enough that I wrote a Perl module named Unknown::Values  .  I proposed adding this new, three-value logic, or 3VL (true, false, unknown), to Perl and while there was some interest, some practical problems reared their ugly heads and the idea went nowhere.  Interestingly, one of the counterpoints was based on the confusion between classical and constructive logic. Complementing undef with a new unknown keyword (that salary might return):  # using undef values: my @overpaid = grep { defined $_-&gt;salary &amp;&amp; $_-&gt;salary &gt; $cutoff } @employees; # same thing with unknown values: my @overpaid = grep { $_-&gt;salary &gt; $cutoff } @employees;  As you can see, using unknown values instead of undef values makes the code shorter and easier to read. unknown salaries would not pass the grep. You don’t have to remember the defined check. It just works . Of course, you can run it in the other direction:  my @underpaid = grep { $_-&gt;salary &lt;= $cutoff } @employees;  And again, unknown values wouldn’t pass the grep. If you need those employees:  my @unknown_salary = grep { is_unknown $_-&gt;salary } @employees;  But this was presented as a counter-argument:  if ( $x &lt; 3 ) { ...; } else { ...; }  With 3VL, surely the else would also have to be skipped because we don’t know if the condition is false? Thus, we could have an if\/else block where the entire construct is skipped. That would be confusing madness!  Except that in 3VL, if the if condition doesn’t hold, the else can still fire because it represents an unknown value, as shown in the $x = {} example above.  The counter-argument might have worked for Java:  if ( someVar &lt; 3 ) { System.out.println(&quot;Less Than&quot;); } else { System.out.println(&quot;Not Less Than&quot;); }  In the above, if you hit the else block, you know both that:   someVar is a number.  someVar is not less than three.   Thus, Java follows classical logic. A statement is either true or false. There is no middle ground. Perl follows constructive logic, even though we tend to think (and program) in terms of classical logic.  Ignoring the type, for Perl this is more correct:  if ( $x &lt; 3 ) { ...; } elsif ( $x &gt;= 3 ) { ...; } else { ...; # something went wrong }  That’s because until we explicitly make a positive assertion about a value, we cannot know if a statement is true or false. Thus, in Perl, an else block is frequently not a negation of the if condition, but a catch block for conditions which haven’t held.  Again, Perl developers (and most developers using languages with dynamic types) tend to think in terms classical logic when, in fact, we’re using constructive logic. It usually works until it doesn’t.  How Can We Fix This?  brian d foy found my Unknown::Values  module interesting enough that he felt it should be in the Perl core:  Unknown::Value from @OvidPerl looks very interesting. These objects can&#39;t compare, do math, or most of the other default behavior that undef allows. This would be awesome in core. https:\/\/t.co\/xmD8yoEjIK &mdash; brian d foy (@briandfoy_perl) December 17, 2021   His observations match my concerns with using undef in Perl. However, when I proposed this to the Perl 5 Porters  , while some found it interesting, there were some serious concerns. In particular, Nicholas Clark wrote  (edited for brevity):   For this:  $bar = $hash{$foo};  If $foo happens to be unknown , is $bar always unknown ? Or is it undef if and only if %hash is empty?  That behaviour is arguably more consistent with what unknown means than the “always return unknown”.  ...  Does trying to use an unknown value as a file handle trigger an exception? Or an infinite stream of unknowns on read attempts?  What is the range [$foo .. 42] where $foo is unknown?  I think that most logically it’s an empty list, but that does seem to end up eliminating unknown-ness. Hence if we have first class unknowns, should we be able to have arrays of unknown length?   Ovid; It has a high-value win in eliminating common types of errors we currently deal with   And massive risk in introducing a lot of surprises in code not written to expect unknowns, that is passed one within a data structure.  Basically all of CPAN.  ...  There are about 400 opcodes in perl. I suspect that >90% are easy to figure out for “unknown” (for example as “what would a NaN do here?“) but a few really aren’t going to be obvious, or end up being trade offs between conceptual correctness and what it’s actually possible to implement.   Needless to say, this pretty much shot down the idea and these were annoyingly fair points.  In other words, leaking this abstraction could break a lot of code and people will be confused. For example, if DBIx::Class  were to suddenly return unknown instead of undef for NULL values, while the semantics would adhere much more closely to SQL, the behavior would be very suprising to Perl developers receiving a chunk of data from the ORM and discovering that unknown doesn’t behave the same way as undef .  So how would we deal with this? I can think of a couple of ways. Each would use the feature pragma to lexically scope the changes.  The first way would be to change undef to use 3VL:  use feature 'unknown'; my @numbers = ( 1, 2, undef,5, undef,6 ); my @result = grep { $_ &lt; 5 } @numbers; # result now holds 1 and 2  In this approach, any undefined value would follow 3VL. However, if you returned that undef outside of the current lexical scope, if falls back to the current 2VL (two-value logic: true or false).  However, we might find it useful (and easier) to have distinct unknown and undef behavior:  use feature 'unknown'; my @numbers = ( 1, 2, unknown,5, undef,6 ); my @result = grep { $_ &lt; 5 } @numbers; # result now holds 1 and 2, and `undef`  This would require unknown to behave like undef if it left the current lexical scope.  In other words, ensure that the developer who chooses to use 3VL has a tightly controlled scope.  But what do we do with the case of using unknown values as a filehandle or the keys to a hash or the index of an array?  $colors[unknown] = 'octarine';  Currently, for the Unknown::Values module, we have the “standard” version which mostly returns false for everything So sort and boolean operations ( $x &lt; 3 ) are well-defined, but almost anything else is fatal. (There is a “fatal” version which is fatal for just about anything, but I don’t think it’s very useful).  Why Not use Null Objects?  When Piers Cawley  first suggested that I use Null Object pattern  instead of unknown values, I balked. After all, you could have this:  my $employee = Employee-&gt;new($id); # not found, returns a null object if ( $employee-&gt;salary &lt; $limit ) { ... }  That puts us right back where we started because when salary returns undef , it gets coerced to zero (probably with a warning) and the &lt; $limit succeeds. That’s not the behavior we want.  Or maybe salary returns the invocant to allow for method chaining. Well, that fails because $employee-&gt;salary &lt; $limit silent coerces salary to the address of the object (thanks, Perl!) and that’s probably not what we want, either.  That’s when I realized an old, bad habit was biting me. When presented with a solution, I tend to look for all of the ways it can go wrong. I need to look for all of the ways it can go right. The more Piers made his case, the more I realized this could make sense if I can use the operator overloading of Unknown::Values . I could write something like this (a bit oversimplified):  package Unknown::Values::Instance::Object { use Moose; extend 'Unknown::Values::Instance'; sub AUTOLOAD { return $_[0] } }  Now, any time you call a method on that object, it merely returns the object, When you eventually get down to comparing that object, the overloading magic in the base class always causes comparisons to return false and we get the desired 3VL behavior.  We could go further and assert the type of the Unknown::Values::Object and do a -&gt;can($method) check and blow up if the method isn’t there, but that’s probably not needed for a first pass.  But there’s more! In the Wikipedia article about null objects  , they use an example of a binary tree to show how Null objects sometimes need overloaded methods. Using syntax from the Corinna OOP project for Perl  :  class Node { has $left :reader :param { undef }; has $right :reader :param { undef }; }  We can recursively calculate the size of the tree:  method tree_size() { # this recursive method adds 1 to $size for every node # found return 1 + $self-&gt;left-&gt;tree_size + $self-&gt;right-&gt;tree_size; }  But the child nodes might not exist, so we need to manually account for that:  method tree_size() { my $size = 1; if ( $self-&gt;left ) { $size += $self-&gt;left-&gt;tree_size; } if ( $self-&gt;right ) { $size += $self-&gt;right-&gt;tree_size; } return $size; }  That’s not too bad, but if we need to remember to check if the right and left are there every time we use them, sooner or later we’re going to have a bug.  So let’s fix that (note: this example is simplified since I’ll need to create a new version of Null objects just for Corinna).  class Null::Node :isa(Null::Object) { # break the recursion method tree_size () { 0 } }  class Node { use Null::Node; has $left :reader :param { Null::Node-&gt;new }; has $right :reader :param { Null::Node-&gt;new }; }  Because we’ve overridden the corresponding methods, our tree_size() doesn’t need to check if the nodes are defined:  method tree_size() { return 1 + $self-&gt;left-&gt;tree_size + $self-&gt;right-&gt;tree_size; }  In this simple case, this may overkill, but if we start to add a lot of methods to our Node class, we will have a Null object as default, using 3VL, and we don’t have to worry about building in extra methods that we don’t need. (But see the criticism section about null objects for more information  ).  Unknown::Values with Null object support is on the CPAN  and github  .  Caveat  Be aware that the latest version ( 0.100 )of Unknown::Values has backwards-incompatible changes  . In particular, any attempt to stringify an unknown value or object is fatal. That is a huge win because it protects us from this:  $hash{$unknown_color} = 'octarine'; $array[$unknown_index] = 42;  Both of those are clearly bugs in the context of unknown values, so they’re now fatal.  This doesn’t address all of Nicolas' concerns, but it definitely goes a long way to handling many of them.  ","title":"Constructive Versus Classical Logic in Perl","url":"\/articles\/constructive-versus-classical-logic-in-perl.html"},{"body":"    Why Leave Oracle?  Time Has Not Been Kind to Oracle  Added Benefits of PostgreSQL  Starting the Process  Success Criteria  SWOT Analysis  Developing a Migration Plan  1. Audit  2. Planning  3. Migration  4. Testing  5. Adjusting  6. Deployment  7. Rollback  Success!       The glory days before databases. Source    Why Leave Oracle?  Oracle is the fast, beastly sports car that will quickly outpace that Honda Accord you’re driving ... so long as the sports car has an expert driver, constant maintenance, and an expensive, highly-trained pit crew to look after it. Source   A couple of decades ago I was working for a company that was using Oracle on Linux. At the time, this was unusual, though becoming more common. It was a time when open source software was still looked upon with mistrust, so while we told our investors we used Oracle, we didn’t trumpet the fact that we used Linux.  One day we discovered a problem. An Oracle SELECT statement was sometimes returning different results. The query in question was extremely complex, but the results should not have changed as the underlying data was static. After much debugging, the company gave up trying to solve the issue and called Oracle. We were informed that while Oracle could run on Linux, it needed to be Red Hat Advanced Server.  So we switched to Red Hat Advanced Server. And the bug was still there.  &ldquo;No, you need be on version X of Red Hat Advanced server.&rdquo;  So we switched to version X of Red Hat Advanced Server. And the bug was still there.  &ldquo;No, you need be on version X, point release Y, of Red Hat Advanced server.&rdquo;  So we switched to version X, point release Y of Red Hat Advanced Server. And the bug was still there.  &ldquo;That’s a known bug. We&rsquo;re working on it.&rdquo;  Words were said. They were not nice words.  Time Has Not Been Kind to Oracle  In the early 2000s, switching from Oracle to PostgreSQL was a gamble. Today we’ve had several clients making the switch or in the process of it, but what holds them back is no longer “can we trust PostgreSQL?“, it’s “our entire technology stack assumes Oracle.”  And the driver behind the switch? It’s invariably Oracle’s insane costs with no clear return on investment for most companies. But there are plenty of technical reasons, too. Oracle DDL is not transaction safe.   Oracle violates the ANSI standard by often treating empty strings as NULL , causing all sorts of subtle bugs. And Oracle was built in an ASCII world, not a Unicode one, and it shows, painfully.  Added Benefits of PostgreSQL  First, PostgreSQL is generally chosen as the migration target due to its strong compatibility with Oracle and its excellent data handling. Fortunately, it’s now easy to find PostgreSQL admins (as opposed to Oracle admins) in the wild because PostgreSQL is both open source and free, so it’s much easier for a would-be DBA to get experience with the stack. In fact, one of our DBAs commented:   I never met anyone who knew how to use Oracle RAC (and debug its problems) unless they had already endured the hell of a broken Oracle RAC installation at least once ... on someone’s production server.   Cloud support for PostgreSQL is fantastic, with Amazon, Azure, and Google all supporting PostgreSQL.  And because PostgreSQL is SQL-92 (and increasingly, SQL-99 compliant) with an open extension library, its geospatial tooling and JSON support are arguably much stronger than Oracle’s. If you need those tools, you save a ton of money and get a better product.  Starting the Process  As with any project of this scope, you need to start out with a solid plan and that plan starts with a problem statement clearly outlining:   The problem you are trying to solve.  Why you want to solve the problem.  What your proposed solution is.  Measurable success criteria.  A SWOT analysis (or something comparable).   While points 1 through 3 are the most straightforward, it’s worth considering unusual ways of meeting your goals. For example, merely contacting Oracle and explaining that you’re considering leaving Oracle for a more affordable solutions might get enough of a discount on licensing that you don’t feel the immediate pressure to migrate. High-profile companies, or organizations that hook people young (such as universities) can often command hefty discounts.  You may also find that hiring an expert to review licensing is worthwhile. For example, hot standby servers need licensing, but cold standby with minimal use may not. Or you may find yourself paying for too many licenses because you have separate databases which can be merged, or you may find yourself paying for too many CPUs when a more powerful server may be a more cost-effective solution.  There are also many edge cases to consider. One company we know uses Oracle for a public facing application so that they can reassure investors that “they use Oracle”, but use PostgreSQL exclusively for everything internal.  Or there are the cases where clients use Oracle Enterprise applications but find many internal cases where they’re unnecessarily spending on Oracle licenses. The Enterprise applications are often difficult to extricate from, while individual applications still eat up enormous costs for no valid reason.  Success Criteria  However, it’s points 4 and 5 above (success criteria and SWOT analysis) that we find many clients don’t address. For example, the success criteria needs to be measurable and you should be able to address:   Performance  Functional requirements  Schedule  Budgeting  Resource allocation  Minimized disruption   All of those will need flexibility when you encounter surprising obstacles to the migration. Resource allocation is particularly important if you are partway through a migration and find yourself maintaining Oracle and PostgreSQL systems simultaneously. This is where having a dedicated team to oversee the process is helpful because they are focused on successful delivery and are less likely to be pulled away on internal “distractions” that arise throughout the migration process.  SWOT Analysis  A SWOT analysis  is simply a listing of Strengths, Weaknesses, Opportunities and Threats. For example, a clear strength of moving to PostgreSQL is the significant cost savings. Weaknesses may include poor internal knowledge of PostgreSQL. Opportunities can involve the custom data types and DDL-safe transactions, while threats might involve a need for write-heavy multi-master replication.  The above weaknesses or threats can be addressed, but you need to understand them and plan for them up front, with a signicant focus on mitigating weaknesses and threats.  Developing a Migration Plan  The migration plan that you develop will need to be fine-tuned for how your company operates, so a “one-size fits all” plan is not possible to develop. However, a successful migration plan should generally include the following steps:   Audit  Planning  Migration  Testing  Adjusting  Deployment  Rollback   1. Audit  The audit stage starts with identifying every area in your organization where Oracle is used, identifying all licenses and expirations (you may need to renew during the migration process), and how Oracle is being used. We’ve found many clients are merely using Oracle as a data store and use few, if any, stored procedures or functions. This is underutilizing the power of the database but has the advantage of simplifying migrations.  The software audit is a critical portion of the overall audit. Many applications using sophisticated ORMs may find they need few, if any, changes when switching from Oracle to PostgreSQL. However, other applications—particularly those building SQL directly—will need a deeper audit to understand how the migration is to be accomplished.  Further, because the migration process is time-consuming, make sure you’re only migrating what needs to be migrated. Do you have historical data that doesn’t need migration? Do you have tables or stored procedures which are no longer used? You may find it simpler to skip those.  2. Planning  With the primary audit finished, it’s time to develop a detailed plan on migrating to PostgreSQL. We know of one company in Portland, Oregon, which managed to get 90% of their software tests on a large application passing in a single evening. More typical, however, is the case where the “Oracle-only” assumption runs deep. This is where PostgreSQL shines, because its SQL is mostly compatible with Oracle’s.  The planning for most common use cases involves the schema migration, data migration, and software migration. If you have a small schema (and this is almost always a case where PostgreSQL is preferable to Oracle), the schema migration can be done by hand. However, there are many open source and commercial tools which can migrate an Oracle schema to a PostgreSQL schema. Care will still need to be taken, however, especially if you have Unicode data.  3. Migration  The actual migration at the database level will take some time, but it in the software level that you’ll particularly experience grief. Having a robust test is imperative, as is a strong QA department.  One issue we routinely encounter is Unicode. Unicode is problematic for Oracle because it was developed in an ASCII world while PostgreSQL fully understands and supports Unicode. For example, in Oracle, depending on the value of NLS_LENGTH_SEMANTICS , you may find that VARCHAR2(10) holds 10 characters or 10 bytes. If it’s bytes, you cannot store 日本 , because that’s actually twelve bytes! In PostgreSQL, VARCHAR(10) is guaranteed to hold 10 characters , not bytes , so 日本 is just fine. (Note: you can also use NVARCHAR2 in Oracle to enforce character semantics, but NVARCHAR2 assumes a maximum of two-bytes per character  , so you still can’t store 日本 in an NVARCHAR2(10) field).  Another common pitfall is that Oracle often treats the empty string as NULL . This is in violation of the ANSI-SQL standard and causes no end of headaches for developers. For example, in Oracle you can often find cases of outer joins with ambiguous semantics because you don’t know if field you’re asking for was missing (not joined), or is the empty string.  Or one surprising issues we’ve stumbled across: Oracle allows you to declare integers ( INT ), but internally it’s a NUMBER(38,0) and conversion tools generally have NUMERIC(38,0) in the PostgreSQL version of the schema. While there are those who insist there’s no performance difference between the two  , when ported to PostgreSQL, you should use the appropriate integer datatypes and not their NUMBER(p,0) equivalents. We’ve found significant performance improvements by converting NUMERIC(p,0) types to appropriate integer types.  There are many other subtle differences ( TIMESTAMP WITH TIME ZONE , join differences, transactions, function incompatibilities, and so on). The free, open source Orafce  tool module will make much of the pain go away, but not all (and it may prove to be a crutch you shouldn’t rely on). Again, comprehensive automated testing backed by a QA department is helpful here.  4. Testing  While it’s improved dramatically over the past decade or two, testing is the bane of many IT companies. Developers tend to test that their software does what they want it to do, while strong QA teams try to test that their software doesn’t do what they don’t want it to do. The difference is subtle, but it’s the difference between creation and destruction.  A key example comes from your issue and bug tracking systems. If you find that certain bugs are recurrent, this is often the case of a developer fixing the general issue for a specific case. A strong QA department should recognize the general nature of the problem and extend their tests to cover the general issue, not the specific one.  While this problem is not specific to an Oracle to PostgreSQL conversion, it is an issue which can mean the difference between a successful and unsuccessful migration. You will need to be sure that your testing approach is solid and that you’re not just testing that everything works with valid inputs, but that they fail in an expected way with invalid inputs.  But be aware that testing isn’t just about whether or not your software works as expected. Can you restore from a backup? How long does it take for you to failover? What’s your throughput like? Do you have “stealth” applications which assume Oracle and you’ve just switched? Perfectly functioning software doesn’t matter when your infrastructure catches fire.  And please keep in mind that this testing isn’t just a “one off” for the migration. A solid testing plan allows you to ensure that regressions don’t occur in the PostgreSQL implementation as it undergoes further development. Spending this time and money up front will save you time and money later.  5. Adjusting  The migration and testing aspects of your transition will likely need to be repeated more than once. There’s not much to say about the “adjustment” phase other than:   You must have measurable success criteria  Those criteria should be automated as far as is possible  Rejected work must document the success criteria and how to achieve them   However, keep in mind that you must not let the perfect be the enemy of the good. Are you excelling in some areas and under performing in others? Is this an acceptable trade off? Are all success criteria really that important? Remember: your primary goal is to facilitate the Oracle to PostgreSQL migration. You can take full advantage of PostgreSQL’s power later.  6. Deployment  Congratulations! Your migration and testing have worked. You’ve successfully met your success criteria. You’ve made adjustments here and there, but you’re ready to roll things out. At this point, it’s very hard to give specific advice because the details of the your deployment will vary widely from company to company.  This deployment should follow your standard patterns as much as possible to avoid confusion, but it needs to be planned carefully, with care taken to minimize downtime as much as possible (especially if your database-backed applications are public facing). However, because this deployment will likely be more significant than most, it does need to be set up in such a way that when obstacles are encountered that cannot be easily overcome, you can immediately abandon the deployment.  If you’re not using it yet, something like a blue-green deployment  approach is recommended here. Changing live systems is hard and it’s easy to paint yourself into a corner. Instead, build a duplicate of the system on production (containers work brilliantly here, of course), and run your tests against your changes. When you’re satisfied that all is well, point everything over to the new system, being ready to switch it back in a hurry, if needed.  And if you do this on a Friday, you deserve the pain you’re going to inevitably going to face.  7. Rollback  If you’ve followed the steps above and gotten to the point of rollout, you’re probably in a good place but you still need to plan for rollback. Will you need it? Probably not. But if you haven’t planned for it and you need it, it’s like having a fire without fire insurance. Discovering a critical production problem a week after deployment could involve losing a lot of data.  There are a few things to note about the rollback plan.   It must be thoroughly documented  It must be thoroughly tested  It is developed in concert with the migration and deployment plans  You must decide if data loss is acceptable or if data migration is part of the plan   The last point is very tricky. If you have something catastrophic happen, you may want to accept data loss rather than risk having to rollback your rollback because you couldn’t migrate new data to the old system. If a rollback is necessary, it’s like to be spotted right away and not a week later, so you’re unlikely to face this decision, but prepare for it.  Remember that a key flaw of release management is that people tend to expect success, not prepare for failure. By expecting success but planning for failure, you’ll put yourself in a great position.  Success!  Congratulations! You’ve successfully replaced Oracle with PostgreSQL. You’ve saved yourself a lot of money, you have a database that’s easier to manage, and you get to take advantage of many of the powerful features of PostgreSQL.  ","title":"Migrating from Oracle to PostgreSQL","url":"\/articles\/moving-from-oracle-to-postgresql.html"},{"body":"  As every developer learns, it’s easy to get naming wrong. For example, if you have a variable named temperature , but your system handles both imperial and metric, is that temperature in Fahrenheit or Celsius? Getting that wrong could lead to all sorts of problems. A similar situation exists for subroutine and method names. What does the method scale do? Who knows? If it’s named scale_size , you have a much better idea.  But let’s discuss predicate methods in object-oriented code. A predicate method is a method that returns true or false. In Ruby, you can end a method name with a question mark and by convention, it should return true or false.  class Point @@number_of_points = 0 def initialize(x,y) @x = x @y = y @@number_of_points += 1 end def self.too_many? return @@number_of_points &gt; 2 end end point1 = Point.new(3,3) point2 = Point.new(4,0) puts Point.too_many? # prints false point3 = Point.new(4,8) puts Point.too_many? # prints true  This is a lovely feature because writing if Point.too_many? reads like a question. However, most popular languages tend to only allow alphanumerics and underscores in method names. So in a recent code review for our free-to-play game, Tau Station ( Sign up here! It’s fun.  ), I called out a case where a predicate method was poorly named. Sadly, it was a predicate method I had written a long time ago and it’s a mistake I still make today. The following examples are made up, but keep to the spirit of the issue.  Quick: what does the following bit of code do? Make a guess. vip stands for “very important person” and means the character has used a VIP pack and they receive in-game benefits.  if ( $character-&gt;vip ) { # do something }  Does vip return how much time is left on their VIP status? Does it return a VIP pack? Does it return a boolean indicating if the character has VIP? We can rename that function to make it very clear.  if ( $character-&gt;is_vip ) { # do something }  Now it’s crystal clear what that means. Predicate functions should generally start with a verb that implies true or false: is_ , can_ , has_ , and so on. So it was with a bit of surprise on a later code review that I read two developers disagreeing about a predicate method name. One named it can_purchase and the reviewer was suggesting allow_purchase . They didn’t budge in their respective positions and both are very good developers, so I was asked to take a look.  In glancing at the code snippet, I immediately agreed that the predicate method should be named can_purchase . But after I wrote that on the review, I had doubts because the answer was too obvious. Why on earth would their be a dispute about this? I decided to dig further. The problem stemmed from an unfortunate feature provided by Moose  (an object-oriented framework for Perl) and perhaps a subtle misunderstanding of object-oriented code.  This is worth an explanation of the confusion.  The code in question was providing an API response for purchasing a visa to visit Gaule-affiliated stations  . The API returns actions you can take regarding a purchase. However, if the character meets certain conditions, the API call should provide additional information allowing a purchase of the visa, including its duration and price.  The code sort of looked like this:    package Veure::Model::API::Visa; use Veure::Moose; use Veure::Types qw(Bool); use Veure::Util::Data qw(:json); with qw(Veure::Model::API::Role::Composable); has can_purchase =&gt; ( isa =&gt; Bool, required =&gt; 1, ); sub _build_api_content ($self) { my $character = $self-&gt;viewer; my %api_response ( area_actions =&gt; { renew_visa =&gt; { available =&gt; json_false }, purchase_visa =&gt; { available =&gt; json_false }, }, ); if ( $self-&gt;can_purchase ) { # add additional data } return \\%api_response; } __PACKAGE__-&gt;_finalize;  That’s the code that I glanced at which made me think that can_purchase is a perfectly acceptable predicate method, until I realized the mistake I made.  Let’s take another look at can_purchase .  has can_purchase =&gt; ( isa =&gt; Bool, required =&gt; 1, );  In the Moose OO framework, the above creates a method named can_purchase . In our version of Moose, that method is read-only, it returns a boolean, and it’s required to be passed to the constructor. So far, so good.  However, the interface for the module is called in a very standard way:  return Veure::Model::API::Visa-&gt;new( viewer =&gt; $character, can_purchase =&gt; $can_purchase )-&gt;api_content;  The can_purchase data isn’t intended to be exposed to the outside world. The method should have been written like this:  has can_purchase =&gt; ( isa =&gt; Bool, reader =&gt; '_can_purchase', required =&gt; 1, );  I’ve written before about needing to keep interfaces small and here we were, violating that rule. Unfortunately, Moose, by default, provides public methods for all instance data and it gets annoyingly hard to avoid this. It’s annoying enough that I often don’t even call that out on code reviews.  This should not be a predicate method. Instead, this is a constructor argument that directs the instance on its behavior. It’s an order , giving a command to the instance. By convention in our code, we like to write our constructor arguments such that orders use active verbs to indicate what’s going on. This is why the reviewer suggested allow_purchase .    This might seem petty or trivial, but the distinction is important because not making said distinction can lead to subtle errors in reasoning. In this case, a predicate method returns a boolean indicating something about the state of the object. It’s only doing a single thing. But objects aren’t just about state. They’re state plus behavior . Predicates return information about the state after the behavior has occurred. This constructor argument tells the object to do something before the behavior has occurred. They look similar, but they’re not.  Because Moose makes everything public by default (because it’s hard not to do this in Perl), including creating methods for your instance data, it’s natural that can_purchase seems like the correct name since it’s public and asking if the character can purchase a visa. But the consumer is making an API calls and receives a data structure, not the instance itself. The data it receives tells the consumer what information to present, not a predicate method call.  I expect that when the Corinna OOP system  finally becomes core to Perl, much of this confusion will go away.  class Veure::Model::API::Visa :does(Veure::Model::API::Role::Composable) { use Veure::Util::API qw(format_price); use Veure::Util::Data qw(:json); field $allow_purchase :new; method build_api_content :private () { my %api_response ( area_actions =&gt; { renew_visa =&gt; { available =&gt; json_false }, purchase_visa =&gt; { available =&gt; json_false }, }, ); if ( $allow_purchase ) { # add additional data } return \\%api_response; } }  In the above, $allow_purchase is not exposed in the interface and it’s much clearer what its role is.  ","title":"Naming and Object-Oriented Code","url":"\/articles\/naming-and-object-oriented-code.html"},{"body":" Nothing profound in this post. It's just a bit of a rant.  As of this writing, there is an advertisement for a \"Senior Perl Developer\" that is part onsite in the Bay Area of the US. The salary listed works out to to about $15 an hour. For those unfamiliar with buying power in the US, $15 an hour is what many politicians are arguing should be the national minimum wage. This company is offering senior software developers a fry cook's pay in one of the most expensive locations in the United States.  At least they're up front about it. The worst job offer I received came as a shock from someone who owned a company well-known in their field, and a company I very much wanted to work for. My previous employer announced they were going out of business and I was scrambling to find work in a serious recession in Portland, Oregon.  I scored an interview with the owner of a small, but profitable consulting firm. They were working with some technologies I very much wanted to work with, but at the beginning of the interview, I discovered that most of their development was in C. I didn't mind that at all, but I confessed that I hadn't touched C in years. \"No problem,\" I was told. He knew my background, had solid recommendations, and was sure I would have no issues picking up C again.  OK, that's a good sign.  Later, he was showing me a propriety programming language they had devised, targeting a particular industry. While showing me code examples, I noticed something strange. I asked the him \"do you know what an SQL injection attack is?\"  As it turns out, they had been vulnerable for years but only discovered that two weeks earlier. He was impressed. And then I noticed their sessions apparently used sequential ids to determine the user, an even easier attack vector! And they had no tests covering this behavior! And, and, and ...  I was getting excited about the amount of value I could bring and wanted to dive in immediately just to fix all of the security holes.  Yes, the interview was definitely going well.  The interview starting covering personal ground and we started talking about politics, food we love, and other things. This was rather unprofessional but happened naturally. Turns out we had quite a few things in common.  I was definitely breathing a sigh of relief here. The dice just kept falling my way and I was feeling like the luckiest many alive.  Then the offer arrived.  It started with a massive pay cut, but given that I was going to learn several technologies I wanted to get into and I was about to be unemployed, I could deal with that, albeit reluctantly. I could pay my bills on it, but was definitely going to have to cut some expenses.  Then I found out that there was not only no paid vacation, there was no vacation.  Um, excuse me? \"We're a small company and we can't afford to have anyone take vacation. However, we need you to work overtime and when you accrue enough, we give you compensating time off.\"  The owner was shocked when I declined the offer. He thought the fact that we got along so well would count for something. Even when I pointed out the massive pay cut combined with overtime and no holidays, he said \"that's just how being a small company works.\"  Sadly, that is how much business works: how much can we get for how little? But there's no employee loyalty when they know you're screwing them. I stuck to my guns and found a much nicer position handling Big Data in Perl. Even when desperate, acting desperate is rarely a good idea.  ","title":"The Worst Job Offer","url":"\/articles\/the-worst-job-offer.html"},{"body":"    The first Corinna commits  What’s Next?  Perl 7  Perl 8  Installation Manager  Concurrency  Data Constraints  Miscellaneous  Conclusion     The first Corinna commits  Recently, some of the first commits of the new Corinna object system  for Perl have been merged into the Perl core. If you’re not familiar with it, I wrote a brief tutorial  .  Around the same time as those commits were merged, I was rummaging around on an old computer and discovered, to my shock, that I had an old implementation of what I was trying to do, before that fateful day at a conference when Sawyer X, the man driving Perl development, saw my work, liked it, but urged me to to stop the implementation and just design something great . (As an aside, it also doesn’t quite appear to match my previous descriptions of it. Oops).  With my initial commit of that code on September 2, 2018, the Corinna project has been roughly four and a half years from conception to the initial core implementation. Since that time, I worked on design, made an initial announcement, and led a design team of some of the top experts in Perl, with much feedback coming from others who’ve designed object systems such as Moose or Moo, and from that effort, Corinna was accepted and the Perl core will have clean, mature OOP system for the first time (and one whose prototype, Object::Pad  has been running in production at more than one company).  Keep in mind that bless and @ISA will still be there. We’re not taking anything away because the Perl 5 Porters (a.k.a., P5P) strive hard to not break backwards-compatibility.  If you’re curious, here’s what a simple 2D point class looked like;  class Point { has [qw\/x y\/] =&gt; ( default =&gt; sub { 0 } ); sub clear ($self) { $self-&gt;@{ 'x', 'y' } = ( 0, 0 ); } }  That’s not good at all, but I was trying to work within Perl’s limitations. It was also largely a response to improve upon Moxie  :  package Point { use Moxie; extends 'Moxie::Object'; has x =&gt; ( default =&gt; sub { 0 } ); has y =&gt; ( default =&gt; sub { 0 } ); sub x : ro; sub y : ro; sub clear ($self) { $self-&gt;@{ 'x', 'y' } = (0, 0); } }  I quite liked the intent of some of the ideas in Moxie, but that syntax is what pushed me to finally build Corinna. By contrast, here’s the same code in Corinna:  class Point { field ( $x, $y ) :param = 0; method clear () { ( $x, $y ) = ( 0, 0 ); } }  There will be an adjustment period while people learn a new way to program in Corinna, the same way that learning a new programming requires learning to think a different way, but overall, the Perl community is very happy about the project. Some are not, but that’s OK. Having a perfectly homogeneous community, marching in lockstep, no questions asked, would be a creepy thing. It would also be a useless thing. Diversity of opinion is not only healthy, but needed.  What’s Next?  That’s a great question. I’ve co-written one book on Perl and written another (linked in the images at the top of the page), I served two decades at The Perl Foundation , the last several years as a member of the board, I have a bunch of modules on the CPAN  , and our company, All Around the World  , is still doing well, with multiple contracts. I’m currently having fun building an OpenAPI  system using Dancer2  (something I’ve not had much chance to play with before), and another company inquired about help from us to go in and fix their object-oriented mess, including cleaning up a bunch of old code to use Moose  .  So we’re doing fine, but whither Perl? Or to abuse Shakespeare:  To Perl or not to Perl, that is the question: Whether 'tis nobler in the code to suffer The slings and arrows of outrageous criticism, Or to add commits against a sea of troubles, And by rebasing, end them? To die(), to sleep(), No more; and by a sleep() to say we end The heartache and the thousand natural shocks That Perl is heir to. 'Tis a consummation Devoutly to be wished. To die(), to sleep(); To sleep–perchance to dream–ay, there's the 'no strict.'  Sue me. I like poetry. Why do you think my nickname is “Ovid”?  Let’s be honest, Perl isn’t as popular as it used to be. There’s a lot of angst in the Perl community expressed over this. There’s also a lot of finger-pointing. There are many reasons that Perl is not as popular as it once was, but rather than ruminate on history and stir up debates over things we cannot change, I want to look to the future.  Fix the problem, not the blame.  So what’s the problem?  Sure, we can talk about fewer Perl developers or fewer companies starting new projects in Perl, but those are symptoms . Instead, I want to focus on one problem that Perl has and I think it’s both important and solvable.  Perl 5 was released in October of 1994 and there’s a misconception that Perl hasn’t changed much since then. I still remember a Linux conference in Prague where a programmer was telling me that Perl hasn’t been updated in decades and this is a widespread misconception that hurts Perl (we suck at marketing).  Just looking at the stable releases in the last decades:        Pumpking  Version  Release Date      Ricardo  5.16.0  2012-May-20    Ricardo  5.18.0  2013-May-18    Ricardo  5.20.0  2014-May-27    Ricardo  5.22.0  2015-Jun-01    Ricardo  5.24.0  2016-May-09    Sawyer X  5.26.0  2017-May-30    Sawyer X  5.28.0  2018-Jun-22    Sawyer X  5.30.0  2019-May-22    Sawyer X  5.32.0  2020-Jun-20    Sawyer X  5.34.0  2021-May-20    Ricardo  5.36.0  2022-May-27     As you can see, Perl gets a regular, major release every year. Some of the new features introduced into Perl include native try\/catch blocks. Function signatures are now on by default. Unicode 14.0 is supported. True booleans are supported. Some older misfeatures, such as indirect object syntax and multidimensional are disabled (they’re still available, but if you don’t know what they are, don’t worry). There are some amazing improvements to the regular expression engine, widely considered the best in the programming world  .  There’s much more, but many issues are very Perl-specific and even many Perl programmers are not aware of the work done by the porters. The Perl 5.36.0 release alone represents approximately a year of development since Perl 5.34.0 and contains approximately 250,000 lines of changes across 2,000 files from 82 authors (taken from the perldelta file for 5.36.0).  Note that 82 developers might seem like a lot, but it’s also incomplete. Many people who’ve contributed haven’t done so by committing directly to the Perl repository.  So Perl is still under heavy development, but it’s hard to argue that we’ve done a great job communicating that, so how can we communicate that?  Perl 7  The excellent Raku Programming Language  was known as “Perl 6” for many years, but it diverged far enough from its Perl roots to merit a separate name. So when there was much excitement about a new release of Perl called Perl 7, there was great news coverage of it  .  In short, even with a fresh Perl install, to use Perl effectively today, you often get the behavior of an older version of Perl and you have to ask for the modern features. If you’re new to Perl, there’s good chance you wouldn’t know that.  Unfortunately, that work was halted amidst disagreements over direction and the Perl Steering Council (PSC) later released a discussion of about what it would take to get to Perl 7  . Is Corinna going to be part of Perl 7? The PSC recently announced that it might  . I think it should, but asking my opinion is like asking a parent if their newborn is cute, so you can ignore me.  But on the off chance you don’t want to ignore me (you’re reading this, right?), Perl 7 will likely be a newsworthy release, showing the world that Perl is still in active development, but the features in the language are rather marginal in value, in the eyes of many. Corinna is the single-largest change to the Perl language in three decades, and that might make a splash, but again, I’m biased. Nonetheless, releasing Perl 7 is not trivial and Corinna might not be the top priority of the developers who will actually get the work done. I can live with that.  At the risk of offending a few people, barring Corinna, any release of Perl 7 is likely to be much of what we have in Perl right now, but with easier defaults. It will simply make it easier to write Perl, but not offer much “new” to current developers, beyond a message to the programming world.  For technical and political reasons,   Perl 7 without Corinna might be exactly what Perl needs. But even then, Perl 7 will be a far cry from the Perl released back in 1994. Perl’s been evolving rapidly over the years, but without people hearing about it.  But how can we push this further? That’s what I’ve been thinking about.  Perl 8  Perl’s regular expression engine is world-class, but it’s hardly the reason to choose a language. Or to keep it. What else does Perl offer? With Corinna, Perl will have one of the nicest OOP systems of popular dynamic languages (er, I may be biased, but when I look at other “popular” dynamic languages, I wince). So that can be two killer features. But we need more, but to get to “more,” we need to address the fact that the Perl core is missing a few features that other languages provide. We need to fix that.  Installation Manager  Recently, I hit an issue with a client. My code was correct, but failed in the test environment due to a “Can’t locate object method” error. As it turns out, the actively maintained CPAN module I was using was last updated in 2022, but the version the client had was released over two decades ago! Worse, my attempt to locally debug this was frustrated by some particulars in the client setup which caused me to find a current local version of the module and not the archaic version the client had. Fortunately, it was a trivial fix, but spending time debugging this issue cost our client money they should not have had to spend.  Per the nodejs blog  :   CPAN and RubyGems have blurred the lines between development tools and system package managers. With npm we wish to draw a clear line: it is not a system package manager. It is not for installing firefox or ffmpeg or OpenSSL; it is for rapidly downloading, building, and setting up Node packages. npm is a development tool. When a program written in Node becomes sufficiently mature it should be distributed as a tarball, .deb , .rpm , or other package system. It should not be distributed to end users with npm .   Out of the box, Perl has a cpan client which lets you install Perl modules. It’s great for developing libraries, but we don’t offer great support for developing applications . When I cd into a client directory, I want to run something like perl-local . and have it pick up the version of Perl for this project and use this project’s CPAN modules . Having this completely isolated from other Perl projects which might have different needs. We have many Perl tools designed to solve various parts of the problem, but it’s time to put them into the core. There’s now a discussion on the P5P mailing list about addressing this  .  To put this as a simplistic user story:   As a developer  I want to easily set up multiple isolated Perl environments  So I can support multiple isolated clients\/projects  Because my work shouldn’t have the fragility of picking up the wrong dependencies   There’s a lot of work there, but a developer installing Perl for the first time should be able to solve this, just as developers downloading other languages for the first time can.  Concurrency  &lt;snark&gt;Promises, promises.&lt;\/snark&gt;  Much of the programming world today is dealing with concurrency in some form. There are serious problems this solves, including:   CPU-intensive work can be split across multiple CPUs (or GPUs!).  Don’t wait on code that might block due to external resource constraints.   P5P has long preferred that new ideas for Perl be tested on the CPAN before incorporating these ideas into the core. This allows us to see what is and isn’t working (or in some cases, just to better understand the syntax people prefer). There are multiple attempts to solve the concurrency issue on the CPAN and given that this feature is being used more and more, and becoming a default feature in many languages, I think the core needs to support his.  Data Constraints  As with an installation manager and concurrency, the CPAN offers many solutions for constraining your data. You might notice I avoided the word “type.” Computer scientists have reasonable disagreements about types and type systems. Computer programmers have shouting matches.  I know what I want from types. Others strongly disagree. However, a very workable compromise has emerged in Perl. This is largely based on Type::Tiny  and similar modules. For Moose, for example, I can define an attribute like this:  has expires_at =&gt; ( is =&gt; 'rw', isa =&gt; InstanceOf ['DateTime' ], );  With that, I can read and write the expires_at value and know that the value must be a DateTime instance or a subclass (er, or anything that overloaded the isa method to lie and say it’s a DateTime class, but we’ll ignore that because it opens up a fascinating discussion well beyond the scope of this post).  For a subroutine, I might do this:  sub reset_totals ($count, $totals) { state $check = compile(Int, HashRef[Int]); ($count, $totals) = $check-&gt;($count, $totals); $count = constrain($count); foreach my $key (keys $totals-&gt;%*) { $totals-&gt;{$key} = $count; } }  Unlike the attribute approach, the sub solution only guarantees the kind of data you have in variables at the time of the $check-&gt;() . In the above subroutine, if constrain returned an array reference instead of an integer, the above code would not only continue to run, but it would do so without warning, to the point of corrupting the data in the code calling this function!  Note that the above is not a hypothetical problem. It happens, but we kid ourselves that our test suite will protect us. In reality, attaching a data constraint to variables instead of statements would fix the above. TypeScript isn’t perfect, but I’ve enjoyed the fact that many of these errors simply don’t occur (though TypeScript handles this differently).  We are not, however, there yet. Type::Tiny and related tools are great ways of minimizing the issue, but their syntax is clumsy and their features are limited, largely because they’re working within the limitations of Perl. This is not a criticism of these modules, though. We have to start somewhere. However, our company  is finding more and more clients relying on Type::Tiny due to the safety it provides. More work is needed here.  Miscellaneous  I think Perl needs native exceptions. It needs native stack traces. Native constants, including deep structures. Copy-on-write would be useful, but it’s not clear how feasible this is. There’s a lot more, but what I’ve written might already be controversial enough that stopping is probably good.  Conclusion  We need a vision of Perl’s future. Many of the things we need in the Perl core are things we’ve re-implemented many times on the CPAN. Not having them in core means developers have to learn their current environment’s way of handling this, and makes it easier to choose a module that might be inappropriate, or worse, unmaintained. This increases the cognitive burden on all developers who, I’m sure, are more interested in building solutions than arguing over them.  Perl 7 is going to be wonderful, but it’s time to start thinking about Perl 8.  If you have ideas about what Perl needs for the future, please leave a comment below.  ","title":"The Future of Perl","url":"\/articles\/the-future-of-perl.html"},{"body":" The following is a true story, but with names changed.  When I work with clients to build software, I take the usual steps of understanding their needs, gathering requirements, learning about their customers, and so on. At this point I have a model on paper of roughly what the software is intended to do, so they get surprised when I immediately turn to database design.  \"Who care about database design? What about mockups? What about workflows?\"  Let me tell you about \"Bob's Luxury Goods.\" I worked for this company many years ago and they had a retail store selling ... you guessed it ... luxury goods. They'd ask all customers for a billing address and if they had a different delivery address. At the database level, they had a \"one-to-many\" relationship between customers and addresses.  That was their first problem. A customer's partner might come into Bob's and order something and if the address was entered correctly it would be flagged as \"in use\" and we had to use a different address or deliberately enter a typo. Fortunately, addresses were case-sensitive, so many people had UPPER-CASE ADDRESSES.  We should have had a many-to-many relationship between customers and addresses so we could handle the case where more than one person would share the same address, but we didn't. Further, I was never allocated the time to fix the database because it was \"cheaper\" to remove the restriction on \"flagged\" addresses and allow a duplicate address to be used.  Naturally, being a luxury goods company, we had many repeat customers and sometimes they would move and if we didn't find the duplicate address, or the address with the \"typo\", we might update the address for one partner, but not the other. That was a headache, but it didn't happen frequently enough for management to worry about it.  That's when the marketing department had a brilliant, inexpensive idea. You see, we periodically did mass mailings of special events to our customers. Since we had the software to do mass mailings, why not import a mailing list of all addresses in high net worth areas and mail everyone about upcoming special events? So the company went ahead and bought a database with all of these addresses, but forgot to mention to me that I was supposed to implement this.  Except that every address record had the customer id embedded in it, so we couldn't enter an address without a customer.  \"Curtis,\" they said, \"just enter a dummy customer called 'Occupant' and attach all addresses to that.\"  Except you couldn't enter a customer without an order.  Except you couldn't enter an order without at least one item on it.  Except you couldn't enter an item unless it was listed in inventory.  Except that reserved the \"inventory\" item and made it unavailable.  Except, except, except ...  It came down to trying to create a fake customer, with a fake order, with a fake item, with a fake item category, with a \"paid\" invoice, with exceptions sprinkled throughout the codebase to handle all of these special cases and probably more that I no longer remember.  Then, and only then, could I write the code to provide \"generic\" mass mailings. Management decided it was easier to hire an outside company to handle the mailing list for them.  If they had simply had a proper database design up front, they could have reused their existing system with little trouble.  That's what bad database design costs you and why I usually start with that before writing my software.  Note: if you're not familiar with database design, here's a talk I give where I make it fairly simple to understand. I mostly avoid big words.  ","title":"Death By Database","url":"\/articles\/death-by-database.html"},{"body":"    Celebrating Ten Years  What do you do?  What makes you different from other consulting firms?  What do you bring to your clients?  Rescuing Legacy Code  Network Provisioning Automation  Data Management  Security  A\/B Testing  What are you the most proud of?  What have you learned during the past ten years?  Trust our instincts  Hiring is everything  Never give up  Why are you developing a game?  What would you like to say to your future clients?     Celebrating Ten Years  All Around the World  is now celebrating its ten-year anniversary. Managing director Leïla Contraire and I collaborated on this post to talk about what we’ve accomplished.  It has been an impressive journey working with American, European, and British clients with an international team. Stabilizing, modernizing, developing new and old code, supporting, training, managing on-site teams.  And that’s not all, we have been expanding.  While at heart we are a Perl consulting company, we have branched out into a variety of languages. We’ve done C++, Lua, TypeScript, Go, Python, Java, React, and more. We still have Perl as a core focus, but being able to diversify not only makes AAW stronger, but gives us the ability to appreciate different approaches to problems.  So after ten years we thought it would be a great time to answer the questions that people at conferences keep asking us.  What do you do?  Our job is to solve a large variety of concerns for our clients: untangling an old code base, handling the work in-house teams don’t have time for, using our expertise to develop new tools that the company has less experience with.  Old or new, there is always this excitement of being able to bring a solution. Given our many years of experience, we have seen just about everything. It doesn’t mean the work is easy—it often requires hard effort both from us and the client—it just means that no matter what is thrown at us, there is this deep certitude that we can solve it and that’s an amazing feeling.  What makes you different from other consulting firms?  In our experience when clients need a consulting company, they either want devs who are going to deliver the company strategy, or they need people who can put together a solution that will address all their concerns.  Because we care, we excel at the second. We love to come in, listen to the client concerns and aims, and propose the best solution, taking into consideration all the parameters. We love to bring flexibility in our solutions, always keeping in mind that we are not here to stay. We are just a temporary booster for a long-term adventure.  We can achieve that because our teams are special. They share our values. They are dedicated, talented, and complementary. Working with them is like being part of an orchestra. Everyone is essential, everyone is part of something bigger than themselves, and seeing them work is simply magic.  In a way to go back to your previous question, we are bringing magic to the tech world.  What do you bring to your clients?  What we bring depends on what the client needs.  Rescuing Legacy Code  Sometimes it’s untangling a mess of legacy code, transforming an old system into something maintainable. After understanding the code, we design an appropriate architecture and gradually, layer-by-layer, transform the code to that architecture. Sometimes we finish that work. Other times we evolve the code to the point where the client is capable of taking over.  Network Provisioning Automation  Of course, we also build new tools. In one case, a transatlantic telecom client was struggling to manage the costs of network provisioning due to it being a manual process. Many large projects were stalled due to time spent managing day-to-day tasks. We built a system that automated the provisioning, including security, freeing their network engineers to work on the difficult problems that actually required human intervention.  Data Management  Data management is another key area. We have found that even the best software developers often do not understand how to design a database or structure large amounts of data to be useful to their company. This often leads to complex, slow code that tries to work around the limitations of their data. We get asked to sort this out. This is often a delicate process, requiring extensive testing as we slowly restructure the data and processes to deliver the quality the client requires.  Security  Security and reliability are uppermost in everyone’s mind today, but they’re often viewed in terms of cost instead of profit. As a result, they often get ignored until the company’s hand is forced.  For one of our largest clients, most of their internal employees had full access to everything on their system. In redesigning their access control they found it both fragile and difficult to implement on a multi-million line codebase. We were brought in and over the course of two years, we deleted large swaths of their codebase, replacing it with a single configuration file and a handful of supporting modules. The project was a great success and made it trivial to have fine-grained control over access, even down to individual components on a page.  A\/B Testing  Today, much of the software world has agreed that testing their software’s behavior is important. But what about testing their customer’s behavior? Does your new sign-up form increase or decrease the number of sign-ups? Can you prove it? One of our customers could, but the A\/B testing system they built was difficult to work with. Because they put all of the data into a single database table, the system required custom SQL be written for every A\/B test. The SQL had to be run from the command line to produce reports.  We rebuilt their entire system with a properly designed database, along with beautiful front-end charts showing the performance of all tests for whatever confidence level desired, including the ability to set the minimum number of required visits per test. Creating a new test only took a few few clicks. Stopping and restarting tests, including adjusting the number of buckets was trivial. A\/B testing is a fantastic tool for discovering if your changes have the impact you’re looking for. You can stop guessing and start using a data-driven approach.  What are you the most proud of?  The certitude that, as a team, we can address any problem that is given to us.  What have you learned during the past ten years?  There are three key lessons we have learned.  Trust our instincts  If something is not right for us, it’s likely not right for the client or our teams. We know when to say “no.”  Hiring is everything  It is not just about assessing technical abilities, but also evaluating communication skills, critical thinking capacity, and the candidate’s desire to contribute to something greater. At our company, we take immense pride in a job well done.  Never give up  Even in the face of adversity, we believe there is always an opportunity for something more significant, happier, and worth building, just around the corner.  With these insights, we are better equipped to steer our company towards sustained growth and success.  Why are you developing a game?  Because we needed to create something we believe in, that belongs to us, to our teams, something that lasts. We are immensely proud of our work on Tau Station  .  As for lasting, it has 8 years of development, and we are only approaching the marketing phase.  It has been a labor of love, the sheer joy of using our experience to tell meaningful stories, and the real struggle to make it happen.  Creating something—giving life to something—is such a different adventure than consulting. It gives us perspective, understanding for our clients, as even in our code base, we sometimes see the pain of legacy code.  What would you like to say to your future clients?  Come and talk to us  . No matter the problem, the need, we have the solution.  ","title":"All Around the World: Celebrating Ten Years","url":"\/articles\/all-around-the-world-celebrating-ten-years.html"},{"body":"   Disclaimer  Social Networking is Broken  The Opportunity  The Background  The Problem  Trust  Horror  Broadcasting  The Unholy Trifecta: Trust, Horror, and Broadcasting  The Contenders  Friendster  MySpace  Orkut  hi5  Google Buzz  Yahoo! 360&deg;  Bebo  Google+  Honorable Mentions  Creating A New Social Network  Excellence  Trust, Horror, and Broadcasting  The David to Facebook's Goliath  Building Trust  Open Source  Open Business  No \"real names\" Policy  Community Engagement  Eliminating Horror  Controlled Broadcasting  AI Categorization Pipeline  Profit?  Can It Happen?     Disclaimer  Yes, I know, this is a big post. You're probably not going to read all of it and that's OK. I understand. However, if you know of others who feel that Facebook needs competition, please share this with them. You can't go up against a multi-billion dollar corporation with a tweet or a 3 paragraph blog entry.  Social Networking is Broken    Via Anthory Quintano   Chris Hughes, one of the founders of Facebook, has called for it to be dismantled  . Needless to say this generated a ton of media coverage for one simple reason: social networks are broken. They’re driving us apart instead of bringing us together. You hop onto Facebook and see old friends from school spewing vile accusations about your favorite political party. Your beloved uncle is sharing another stupid meme while you sigh and search Snopes for a URL to post in response. You see close friends in relationships publicly, bitterly, \"break up\" online where everyone can read their accusations. And that's the mild stuff; the hard stuff is much, much worse.  You jump from news stories about Facebook being used to build support for genocide  to foreign propaganda campaigns to manipulate elections  to livestreaming a mass murder in New Zealand  .  And it's not stopping, though Facebook is belatedly responding. They take down the livestreams of the regular rapes  , the gang rapes  , the child rapes  , the murders and suicides  , and more. This is somewhat better than in 2016 when Facebook refused to remove the livestream of a murder because \"it doesn&rsquo;t violate the company&rsquo;s community standards\"  .  And before we go any further, just ask yourself one question: how can a company get so wrong that they think livestreaming murders isn't violating \"community standards\"?  The Opportunity  Let's see if we can figure this out. Facebook has 2.3 billion monthly active users  , millions of whom want anything else, and 15 million of their most valuable customers have walked away since 2017  . Despite this, Facebook still basked in almost seven billion dollars in net income for the final quarter of 2018  . You would think that investors would be drooling at this. Millions of Facebook high-value customers are begging for a new market! Even more interesingly, there's little barrier to entry into this market.   Taking a percentage of Facebook's market share is a huge business opportunity but investors don&rsquo;t seem to be biting. Why is that? And can it be changed?    Guess which one is the 800-pound gorilla? Source   Well, as it turns out, investors were biting but they were also bitten. With seeing so many other social networks getting up, swinging, and missing, it's no wonder that investors today are wary. However, once you start diving into mess, you find that many otherwise savvy investors have fallen for tulip mania  and haven't paid attention to the real issues customers care about. Thus, the contenders fail.  The Background  Many of these would-be competitors have tried to take on Facebook, with the recent Google Plus shutdown being one of the more dramatic failures. Given the long history, we need a lot of backstory to unpack what's going on here.  First, let's distinguish between a General Purpose Social Network, GPSN, and a Special Purpose Social Network, SPSN, (though there's a lot of grey area here). Many of the millions leaving Facebook for Instagram  , are leaving a GPSN for a SPSN. Instagram is for sharing photos and videos. If you have no photos or videos to share, you may not find an Instagram account as appealing, though you may have one to follow others.   Twitter is closer to a GPSN, but it's still rather limited in its features (that, in itself, is a feature). Facebook, like Myspace, or the now-defunct Google Plus, is a GPSN. It has a wide variety of features that appeal to many people. Want to connect but don't want to share photos? Facebook. Want to connect and don't want to be limited to 280 characters? Facebook. Want to connect and regularly organize or participate in local events? Facebook. Want to have a richer reading experience about what your friends are doing? Facebook. Facebook has it all.  GPSNs are more geared towards general-purpose social networking without trying to shoehorn you into a particular mode of sharing. While many people enjoy a variety of social networks, Facebook dominates in terms of daily user engagement  , with half of its users logging on more than once a day.  The distinction between GPSN and SPSN is important because this article is only focusing on the former. The latter gets lots of traction in narrower markets and has a robust, lively community of competitors. GPSNs, however, have been reduced to \"Facebook or nothing\" as of this writing.  The Problem  Before we can really get down to the business case for a Facebook competitor, we need to understand why so many people are clamoring to leave Facebook. Once we do that, we can understand why so many David&rsquo;s have fallen before this Goliath. And once we do understand what is wrong with Facebook, the failures become almost obvious (given 20-20 hindsight).  First, let&rsquo;s be crystal-clear about what Facebook has done right: they&rsquo;ve created a generic product that has everyone with an internet connection as their target market . If you can connect to the net and you want to be social, Facebook is perfect for you. Of course, if you say \"everyone is my target market\" to an investor, that&rsquo;s a quick way to get them to leave, but given Facebook&rsquo;s 2 billion+ user accounts, we really do have a serious \"everyone\" claim here.      We are social by nature. Source   Second, given that so many people on the net want to be social, the second major thing that Facebook has done right is not being absolutely awful at what they do. That sounds ridiculous, but when you look at the clown car of wanna-be competitors, it&rsquo;s amazing how many of them actually got traction and then fell down because they weren&rsquo;t up to the job, technically.  Third, given the fact that the core Facebook product started out rather limited, they&rsquo;ve been smart enough to offer APIs to allow others to build value for Facebook  , something that allows them to offer much more to the public without having to build it themselves. Just look at all of those annoying game requests you get to begin to understand whats' happening here. Surprisingly, many competitors have ignored this.  So what has Facebook done wrong? Just about everything else.  After being relentlessly pummeled by Facebook-related horror stories, I finally said \"enough\" and started really digging in to understand what is wrong with Facebook. I've found three major problem areas that need to be solved, but they generally have the root cause of being a secretive, unaccountable company focused primarily on profits and market domination. As a businesperson, I will humbly suggest that there is nothing wrong with that. Honestly, you want to make money and destroy your competition? That's perfectly fine.  Where I draw the line is ethics . Facebook has such a huge impact on the public sector that excluding itself from accountability and acting in complete secrecy has allowed it to become a monster. Ironically, Facebook has been so horrifying, and there have been so few interesting competitors, that people are suffering \"compassion fatigue\" and when Facebook does something awful, they just ignore it or share it on Facebook. Things that would kill any other company are shrugged off for Facebook.  So I dug in, hard, trying to get to the bottom of this. I've read so many articles about Facebook and their atrocities that I fear I'm becoming numb to them, but that means I can give you a summary and spare you the worst.  Trust  This entire section could simply be about the Cambridge Analytica scandal  . Facebook essentially allowed Cambridge Analytica to mine Facebook's data to manipulate the US public opinion in the 2016 presidential election, and British public opinion on the Brexit vote  . Or there were the Facebook ads purchased by the Russian government to influence the US election.  , even though Facebook originally said there was no evidence for this and stonewalled US Congressional investigations  .  Or maybe you don't like Facebook, so you don't have an account. Don't worry, thanks to \"shadow profiles\", Facebook still probably knows who you are  and there's nothing you can do about it.  Or consider the time when Apple had to revoke a Facebook app's permission because Facebook cheated and signed the app with a certificate they weren't allowed to use.  . There's no excuse for this, but they cheated for the sole purpose of collecting more data about people, including people as young as 13.  And why not consider the incredibly poor security mistakes made when \"protecting\" your data? For example, Facebook stored millions of passwords in plain text  , one of the most basic of security blunders. Or how about discovering that half a billion user records were publicly available after they were uploaded to Amazon Web Services  . Facebook could put an end to that by not letting this information be collected by third-party applications, but that's not going to happen.  As for privacy, as long as Facebook maintains their real-name policy  , you will never have strong privacy guarantees. Facebook has tried to mitigate some of the issues with this policy, but their \"fix\" raised more problems than the policy itself  . Facebook is unlikely to address this issue because fake names cause the share price to drop  .  The privacy issue hits Facebook again and again and again. For many people, it's their number one reason for wanting to find another GPSN. Zuckerberg has flipped from privacy is no longer a \"social norm\"  , to insisting Facebook would become a \"privacy-focused communications platform\"  . Given Zuckerberg's history, he's more likely to be known as \"the boy who cried privacy.\"  Of course, given all of the scandals, why doesn't the board simply fire Zuckerberg as CEO and move in a new direction? Because they can't. Mark Zuckerberg has a majority stake in Facebook and can fire any director, at any time, without cause.  . The board can't lift a finger against Zuckerberg and he knows it. He has complete control over everything Facebook does.  Horror  The Myanmar military used Facebook posts to gain support for the genocide against Muslims in Myanmar  . White nationalists have long used Facebook to coordinate activities, but only recently has Facebook started banning those groups  . And a Kenyan vigilante group is using Facebook to murder people suspected of being in gangs  . These and many other scandals would be enough to destroy many companies, but \"scandal fatigue\" has definitely set in. However, this isn't where the worst of it is.  Facebook's livestreaming is mostly boring, innocuous, and sometimes touching. And every once in a while its mind-numbing horror will turn your stomach. It's incredibly painful to read about.  It includes suicides, parents murdering their children, parents abusing their children, gang rapes, and more. And do not forget the New Zealand mass murder that was livetsreamed.  Facebook (read: Zuckerberg) knows this. Despite repeatedly livestreaming horrific crimes, Facebook knows they are making a lot of money and Zuckerberg refuses to do anything about it.   They've been repeatedly asked to add a time-delay to livestreams, or remove them entirely until they can figure out how to stop this. However, they're serious about not delaying livestreams, even if there are brutal crimes involved, but they're taking a huge amount of heat over this. Their response?   “From now on, anyone who violates our most serious policies will be restricted from using Live for set periods of time — for example 30 days — starting on their first offense. For instance, someone who shares a link to a statement from a terrorist group with no context will now be immediately blocked from using Live for a set period of time,” Facebook VP of integrity Guy Rosen wrote.  Read that again. This is the \"Vice President of Integrity\" explaining that Facebook won't delay the livestreaming of rapes and murders.  Broadcasting  This issue seems relatively minor from a \"news\" point of view, but it turns out that it's one of Facebook's biggest problems. From a source inside Facebook who cannot be named: \"the number one reason people leave Facebook is because their mom joins.\" An independent study confirmed this.  There are several things going on, much of what deals with the difference between online and offline social interactions. Offline, social interactions are face-to-face. We know who we're dealing with and can adjust ourselves accordingly. Online, you're in what I call \"broadcast\" mode. You don't necessarily know who you're communicating with and nobody wants to tell a dirty joke to their grandmother.    Social media broadcasts whether you like it or not. Source   Google+ attempted to address the \"broadcasting\" issue by creating \"circles\" that you could put people into. If you posted, you could say \"just share with these circles\" and Grandma would never see that naughty joke you shared. But it was a chore trying to always remember which content you could share with which circles. People join social networks to be social, not to remember which switches they need to flip every time they want to be social.  And the Google+ circles, while trying to fix what you were broadcasting, did nothing to address what you were receiving. The broadcasting problem works both ways.  The Unholy Trifecta: Trust, Horror, and Broadcasting  So why do I focus on these three problems of Facebook? From my reading, they're the three central problems that Facebook cannot overcome (in this analysis, privacy falls under \"trust\").  If you're going to take on Facebook, you have to address all three of these areas.  This is one of the premier reasons why the various competitors have failed. The software landscape is littered with failed replacements for Microsoft Word because the programmers correctly realized that people only use a handful of Word's features, but they failed to notice that everyone uses a different handful.  I've talked to many people about Facebook, read many articles, and have seen that it's the same problem as replacing Microsoft Word. Many people don't care about the broadcasting issue, or the privacy issues, or livestreaming issues, and so on. Trying to fix just one issue ignores large swaths of people who don't care about your issue. In fact, most competitors seem fixated on the privacy issue when it's the broadcasting issue most people care about!  With that in mind, let's look at our contenders.  The Contenders  What follows is not an exhaustive list of contenders, but it's representative of the major contenders and their failures. The key takeaway here is that some contenders definitely could have taken on Facebook, but failed in almost comical ways.  Friendster  Friendster wasn't the first social network, but it's probably the best-known early contender.  They launched in 2002 and quickly racked up millions of followers, several competitors, and a $30 million buyout offer from Google.  The founder, Jonathan Abrams, turned down that offer and Friendster later failed, slain by MySpace. There were many reasons for this failure, but a critical issue was simply poor performance. As it grew in popularity, its home page could take almost a minute to load.  MySpace  MySpace, once the most popular Web site in the US, spectacularly imploded after Rupert Murdoch purchased them for $580 million.  . A large part of this seems to be that Murdoch's NewsCorp just didn't \"get\" the internet. You move fast and you react to what your users want; post-buyout, MySpace moved slowly and built a bunch of buggy, half-thought out features.  Seriously, who spends valuable developer time and money building a virtual karaoke machine? Meanwhile, Facebook was opening up their API so that third-party companies could build the features for Facebook, allowing Facebook to focus on their core products. So again, we see the death of a social network due, in part, to technical limitations which should never have happened.  Orkut    Orkut in all its glory! Source   Orkut could have become something. It was launched to much fanfare, including this bit from The Register  (emphasis mine):  Undetered by the feeding frenzy around the social networking bubble, and rebuffed by Friendster Inc, which it attempted to buy, Google has decided to build one better. Given Friendster's well-documented problems with coping with a large number of users, and Google's world class expertise in scalability , it ought to be more than up to the technical challenge.  Except ... here's what Marissa Mayer, then with Google, had to say about Orkut  :  Orkut launched in January of 2004, and as Mayer remembers it, it attracted “several million users in a few days“. This caused the network to slow down to a crawl, Mayer noted. And that, in turn, turned a lot of users away from it — at least in the U.S. “It’s all about speed here,” she said.  Vox has an interesting article about the mess that was Orkut  , but it basically amounts to a part-time project, written by Google employee Orkut Büyükkökten, getting co-opted as a Google product, even though it was never designed to scale. What's worse, because it was largely built with Microsoft products, Google engineers deemed it \"not scalable\" and by the time they finally got around to fixing it, Facebook had been launched and the damage had been done. So again, we see the death of a social network due, in part, to technical limitations which should never have happened.  hi5  Not much to say about hi5  . Launched in 2004, for a few years they were the second or third most popular social network in the US. speeli.com has an interesting summary of why hi5 went nowhere.  Amongst other things, you could see who viewed your profile and thus had the incredibly creepy experience of knowing who your cyber-stalkers were. However, their failure was probably due to the fierce competition with MySpace and Facebook, combined with poor earnings and a funding collapse  . By 2009, hi5 was focusing heavily on virtual goods for monetization and while Facebook focused on user acquisition around the globe.  Facebook, of course, had heavy VC funding, and the hi5 social network never quite seemed as polished. Perhaps if hi5's last funding round hadn't collapsed, things would be different today.  Google Buzz  (Note: we're ignoring \"Google Wave\" because, despite people talking about it as being the predecessor to Buzz, it was a collaboration tool and not a social network)  Included only for completeness: Google has a habit of launching various social media products, seeing little user engagement, and then dropping them again. In the case of Google Buzz  , many users didn't understand what it was for, it publicly displayed who you most chatted and emailed with  , and the class action lawsuit probably did little to help.  . It lasted only two years before Google put it out of its misery.  Yahoo! 360&deg;  Remember them? Me neither. Nobody really had any idea what it was for, but it was \"social\"  .  Bebo  Bebo really could have been something. At their height they had over 40 million users with strong engagement and sold themselves to AOL for a shocking $850 million!  AOL, rather than having a sound vision of where they were going, was trying to use the piles of money they were sitting on to buy something popular and stay relevant. Two years later AOL sold Bebo for $10 million.   Oops.  AOL, somehow, someway, got this bizarro idea that they would pay almost one billion dollars for a small social network and squeeze and squeeze and squeeze money out of the users. But that was never to be. They had a host of internal issues and reportedly never invested heavily in Bebo after the purchase. Bebo had great user engagement and a 3rd-party integration API that worked well; they actually had a chance, but there was no way AOL was going to earn back that money.  Google+  Google Wave lasted two years before Google killed it. Google Buzz lasted two years before Google killed it. Orkut hung on for about a decade before Google killed it.    Google+ Launch   And then there was Google+  , arriving at a time when many people were already getting sick of Facebook and hoping for a decent alternative. And then it launched.  It's hard to be sure exactly what Google was thinking, but one designer on the project described an epic tale of fear, politics, and siloed development.  . Vision? The entire thing was driven by fear of Facebook instead of \"what value can we bring to our customers?\"  And then there's Steve Yegge's infamous Google rant  . He meant it to be private, but didn't understand how to use Google+, so it was posted to the world by accident and lovingly copied, again and again, so that it wouldn't be lost if he was ordered to take it down. It can be summed up with one delightful paragraph:  Google+ is a prime example of our complete failure to understand platforms from the very highest levels of executive leadership (hi Larry, Sergey, Eric, Vic, howdy howdy) down to the very lowest leaf workers (hey yo). We all don't get it. The Golden Rule of platforms is that you Eat Your Own Dogfood. The Google+ platform is a pathetic afterthought. We had no API at all at launch, and last I checked, we had one measly API call. One of the team members marched in and told me about it when they launched, and I asked: \"So is it the Stalker API?\" She got all glum and said \"Yeah.\" I mean, I was joking, but no... the only API call we offer is to get someone's stream. So I guess the joke was on me.  Building a social network without an API is shooting yourself in the face. This lesson has been learned again and again and again. But not by Google.  Google+ introduced \"Circles\", allowed you to edit your posts (this competition finally pushed Facebook to add this oft-requested feature), Hangouts, online photo editing, and a few other nifty features, but the lack of an API and a generally \"clunky\" feel meant that Google+ was off to a bad start, despite picking up millions of users.  And then Google said \"hey, let's piss off Youtube fans!\" and required a Google+ account to comment on Youtube videos  . And yes, people were furious. Not only were they forced to have a Google+ account to use Google services, but the \"real name\" policy meant that many people, including those who have a genuine need to hide their identity, were locked out. Google finally gave up on these changes a couple of years later  . I can't help but wonder who at Google thought that force feeding a social network to their customers was really the fastest way to their hearts.  That being said, there were some nifty innovations led by Google+, but at the end of the day, without a clear vision, Google+ slowly faded away until Google put yet another product out of its misery.  Honorable Mentions  In the history of social networking, there are still a few other sites worth mentioning.  One of the most famous is Diaspora  , still in existence as the Diaspora Foundation  . They tried to raise $10,000 to compete against Facebook but quickly raised over $200,000 dollars  , despite not having written a single line of code. They were known as a \"Facebook killer\" long before entering that search term in Google turned up murderers. Sadly, their initial release was riddled with security flaws  , helping to stall momentum. Another serious issue is that, because it's decentralized, ISIS terrorists have started using it because it's almost impossible to censor them there.  Diaspora is asking users to inform on other users who might be terrorists in an attempt to get larger \"pods\" to take down suspicious accounts.  At the end of the day, end users didn't really find a \"decentralized social network\" interesting, or they simply didn't understand it. Diaspora was doomed.  Another interesting idea was minds.com  , the Libertarian-leaning  , crypto-currency backed social network.  Yes, you read that right. Someone thought that a crypto-currency based social network was just what the world needed.   Unsurprisingly, their traffic, never high to begin with, collapsed in the 2018 cryptocurrency crash. Any money you earned through them became instantly worthless. However, they are open source  and should get kudos for that, but it's not enough to save them.  If there is one social network I wanted to like, it was MeWe  , a \"censorship-free\" social network with none other than Tim Berners-Lee  , the inventor of the Web, as an advisor. But guess what happens to sites like Gab, Voat, and MeWe, when they claim they won't censor? They attract those being censored. I'm sure Berners-Lee isn't very happy with headlines like Sir Tim Berners-Lee’s app MeWe is used by neo-Nazis and perverts  :  Sir Tim Berners-Lee's app MeWe included neo-Nazi posts, upskirt “creep ... content, including posts from paedophiles and terrorist supporters.  I went and checked out MeWe myself and found myself in a cesspool of white-nationalists, \"WeTheSheeple\", MAGA-enthusiasts, \"Exposing Satanic World Government\" members, and some, I assume, are good people. It's definitely leaning \"alt-right\" and this Reddit posts summarizes it very well.   Creating A New Social Network  Note: If you have just skipped ahead to this section, this discussion won't make as much sense.  Chris Hughes, in the aforementioned article stating that Facebook must be dismantled, referred to undoing their acquisition of Instagram and Whatsapp. But those are social networks with very specific niches, the SPSNs mentioned earlier. There are still plenty of SPSNs out there. Instead, Facebook killed the GPSNs and, as Hughes stated, gave Zuckerberg a unilateral \"ability to monitor, organize and even censor the conversations of two billion people.\"  Excellence  What's needed isn't to destroy Facebook; that would risk exchanging the devil we know for the devil we don't. Instead, what we need is competition. The various attempts at competition I mentioned earlier failed for various reasons. The reasons that never should have occured are the cases of technical incompetence. Friendster, MySpace (after the NewsCorp acquisition), Orkut (the disastrous launch), Diaspora, and others not mentioned. However, I've been in this industry for far too long for me to suspect that management are going to get better at strategic planning,   or that software \"architects\" will learn to go by evidence instead of their gut, or software developers will suddenly learn how to write clean, extensible code.  If a major attempt is made to take on Facebook, the attack must be well-thought out and there must be no shortcuts on the path to technical excellence. The investors need to understand that building value is more important than saving money&mdash;none of this idiotic \"we can save money by hiring developers for $100 a day!\" You don't enter a mature, multi-billion dollar market by winging it.    Trust, Horror, and Broadcasting  As mentioned earlier, many companies have failed to replace Microsoft Word because they offered a limited subset of features  , ignoring that different users offer different subsets. Many of the would-be competitors did a good job of addressing some of the core issues of trust, horror, and broadcasting, but no one has attempted to address all of them. If you want a GPSN to compete with Facebook, addressing all of these issues up front is going to make life easier, but how do we do that?  Some (Diaspora and minds.com) have attempted to address the trust issue by being open source, but have failed in other ways, such as terrible security issues (Diaspora) or having this weird \"cryptocurrency social network\" thing that simply wasn't appealing to most (minds.com). Others have attempted to address trust by saying \"no censorship\" (too many to list!), and wind up being cesspits for racists, conspiracy theorists, and pedophiles.    No, we don't need to host this content. And yes, this is a real screenshot.   While I believe strongly in freedom of speech, not every platform needs to allow it. But what speech to censor? By law, some speech must be censored (for example, child pornography), and that censorship varies from country to country. And that raises the question of whether or not the new company should give in to demands from authoritarian countries.  The David to Facebook's Goliath  For the sake of having a name, let's call our theoretical company \" Ubuhle  \", the Zulu word for \"beauty\" or \"goodness\".    Many companies, including powerhouses like Google, have tried and failed to take on Facebook. What we need is something radically different, addressing all of the issues above. However, unlike Facebook's relentless drive for profits and dominance, Ubuhle would be focused on the needs of its users. If a social network can be thought of as a public service, the company should act in the interests of the public, not shareholders, or Zuckerberg. Part of that difference would be creating Ubuhle as a B Corporation. B Corporations are different.   Certified B Corporations are a new kind of business that balances purpose and profit. They are legally required to consider the impact of their decisions on their workers, customers, suppliers, community, and the environment. This is a community of leaders, driving a global movement of people using business as a force for good.  B Corporations include Ben & Jerry's, Amalgamated Bank, Kickstarter, and thousands more. You can make a profit and guard your soul at the same time.  Next, let's address the trifecta.  Building Trust  Open Source  Let's just get this out of the way right now: Ubuhle's software should be open source. There are many successful open source companies  and Ubuhle's strength would be the service and trust it provides, not just the code. When the code is open source, people can look to see if we're building shadow profiles, or inappropriately mining data, or using lax security practices.  Open Business  Next: open business. Facebook has taken in a lot of money from questionable sources, such as Cambridge Analytica and Russia advertising propaganda campaigns  targeting US voters. Ubuhle should have a page where users can search for all income sources, along with their details. Instead of shrouding everything in secrecy, let it be out in the open. The default position of Ubuhle is that people have a right to know this information.  No \"real names\" Policy  Next: use any name you want. While Ubuhle would certainly like to see real names because it's easier for people to connect, in reality, many people have excellent reasons to not use their real names. Imagine if someone is being stalked by an ex and doesn't want to be found. Imagine if someone from Iran is gay and doesn't want to be executed. Imagine if someone just doesn't think other people need to know their real name. Imagine tons of celebrities who, technically, can't use Facebook because their stage name doesn't match their real name. Heck, I'm better known as \"Ovid\"  in much of the software world, but I can't use that on Facebook. And trying to enforce a \"real name\" policy is simply fraught with error. Facebook has banned many people for actually using their real names  .  Community Engagement  When Ubuhle makes major business decisions, the decisions, and the reasons for them, should be shared with the public immediately. Comments should be allowed and Ubuhle should directly engage with its customers. There should be no shadowy cabal of management making secret decisions controlling the communications of people.    Eliminating Horror  This one is easy: no livestreaming unless we can figure out a way to make it safe. But we can go beyond that. Community engagement would be necessary to find an appropriate way to decide what content is acceptable. Certainly we don't need to show the exploitation of children. And while expressing support for neo-Nazis may be legal in most countries, Ubuhle doesn't have a requirement to host that content. But only by working with the community can we navigate what are, to be honest, very difficult waters. Free speech advocates may be horrified by suppressing holocaust denial posts, while others may be horrified if Ubuhle allows non-pornographic nudity. You can't win, but you can compromise.  Horror means different things to different people: livestreaming rapes and murders qualify, but so does promoting terrorism, racism, and other forms of hate speech. How would Ubuhle handle that? How would Ubuhle support free speech but provide a \"safe\" environment?  The answer, curiously, may lie in fixing the \"broadcasting\" problem.  Controlled Broadcasting  Nobody wants to tell their grandma a dirty joke. That's why Google+ created circles: you controlled what you shared and with whom. But always having to remember select the circles for posts kind of kills the whole \"I just want to share\" aspect of being social. Sure, when I'm sitting in front of grandma, I watch my tongue. But if I'm out with friends, I don't. On Facebook, you don't know if grandma is listening to what you're telling your friends and many people leave Facebook as a result.  We can fix this with content tagging and you decide the filtering.  A default Ubuhle filter that all new users might get is \"no nudity\", \"no violence\", and \"no heavily downvoted posts.\" Maybe you decide to go one further and remove politics and religion. Maybe someone else wants the politics and not the religion. Someone else might create a custom filter to skip all posts containing \"Game of Thrones\" and \"GoT\", hoping to avoid spoilers.  With this approach, you control what you see, not Ubuhle. While certain content would still need to be banned (child-pornography), you choose your experience. This filter could be applied to all content. With the default Ubuhle filter, assuming white-supremacist groups are heavily downvoted,   you'd never see them when scrolling through groups.  That's what you read, but what about what you write?  Imagine that I've set up three groups:   Family  School friends  Close friends   My school friends are mostly from Texas and while they're a bunch of great people, it's fair to say that we find each other's political views polarizing. I could create a custom filter just for that group, which excludes anything political I may write. Trust me, they would appreciate that.  What if someone's in more than one group? The most restrictive filters would apply, unless you explicitly tag someone.  But how do we actually tag the content? There are a number of ways to do this, but one way might be ...  AI Categorization Pipeline  For a first pass, there are quite a few open source AI solutions which can categorize text, image, and video content. New content would shoot through a pipeline of categorizers which would tag content, but not filter it (aside from anything illegal). Would it be perfect? No, you'd still get the occassional unwanted content. Would it be better than blasting you with any and all content that you may not want? Absolutely. And unlike Facebook's human moderators  , the AI-Pipeline would suffer no PTSD, would not need frequent breaks from work, would not need to go home to sleep, and would be very unlikely to consider a class action lawsuit against its employer.  As an added bonus, a quick Fermi estimate  suggests that one powerful box running non-stop could replace 100 human moderators. Even being off by an order of magnitude, that's a nice financial saving.    Profit?  So how do you make money off of this? Many of the competitors appear to have avoided ads. I think that was a huge mistake. People aren't idiots and they know that a social network needs to make money. Advertising has been around since before recorded history, the first TV ads appeared in 1941  , and Nielsen has been doing targeted advertising since the 1930s  . However, in the internet era, that targeted advertising is becoming invasive, with ads following us from site to site, to our phones back to our desktops.  And it gets worse. Though I doubt this was intentional on Facebook's part, they've been charged with violating the Fair Housing Act because whether or not you could view an ad for housing depended on your race or your skin color.  Facebook managed to settle the issue by ensuring that anyone \"placing housing, job, or credit ads will not be able to target users by age, gender, or zip code.\"  . But why does Facebook get to decide that?  Ubuhle should have advertising, with crystal-clear controls over what information you do and do not wish to share with advertisers.  You can even opt out of the advertising entirely. That's what trust is all about.  But why allow advertising at all? For the final quarter of 2018, Facebook&lsquo;s ad revenue was $16.6 billion dollars.  , with 93% of that coming from mobile advertising.    I'm unsure how they'll earn $16.6 billion in revenue selling emojis.   Meanwhile, Tim Berners-Lee's offering, MeWe, doesn't allow ads but is selling cartoon icon sets at $4.99 each. You do the math.  And let's be honest: well-done ads can be brilliant.     There are plenty of other ways to earn money, but advertising is the key. What would set Ubuhle apart is giving users total control over that advertising.  And if the user wants to use an ad blocker? That's fine. there is no need to waste time or money on an adblocking arms race.  .  Can It Happen?  Yes, it can. Frankly, we would love to do it  , but we have to be realistic. This would be a long-term, cash-intensive project. Between our client work and building Tau Station  , our free narrative RPG, we are not in a position to do this without major funding. If we don't do this, someone should. Hence, we're putting this information out there.  But as Facebook co-founder Chris Hughes made clear in this Time to Break up Facebook  article:  ...would-be competitors can’t raise the money to take on Facebook. Investors realize that if a company gets traction, Facebook will copy its innovations, shut it down or acquire it for a relatively modest sum. So despite an extended economic expansion, increasing interest in high-tech start-ups, an explosion of venture capital and growing public distaste for Facebook, no major social networking company has been founded since the fall of 2011.  Investors are afraid. None of the competition has really done anything particularly different from Facebook, aside from no advertising, no censorship, and no chance. Ubuhle would be establishing a pact with its customers: you get full control and, in exchange, we work with you to build a community of trust.  Building the business would start with targeting the valuable 12 to 35 year old demographic that&lsquo;s leaving Facebook  . It would take a serious marketing campaign, probably combined with an attempt to swing \"influencers\" over to the new GPSN.  Leveraging network effects will also be key. Ubuhle would need a strong API, preferably one that is similar to the Facebook API   to accomodate those developers already familiar with Facebook. Being able to add their apps to Ubuhle would help it grow with less effort. Widgets will also be a key part of this. Having tools on your web site such as \"Login with Ubuhle\", \"Share on Ubuhle\", \"Ubuhle profile badges\", and so on.  And, of course, there's the question of scale. There's a lot I could say on this topic, but that's turning to the technical side instead of the business side, so that's a different conversation. However, it needs to be well-planned because catching bugs in the design phase is one of the best ways to keep costs down  .  None of this is cheap, but the long-term value is what&lsquo;s imporant, not the development costs. . However, it involves a commitment of years to have the chance to offer solid competition to Facebook and no one wants to take them on. But creating competition to Facebook is important, so this is offered as a chance to get discussion going. I spent weeks digging through information to come up with all of this and at times, it was unpleasant work (the livestreaming research particularly turned my stomach). But we need this. Sooner or later, someone's going to take Facebook down. Sooner is better; even the co-founders agree.  Update : One of the engineers who worked on Google+ offers some great insight as to why it failed.   ","title":"How to Defeat Facebook","url":"\/articles\/how-to-defeat-facebook.html"},{"body":"   Building a Remote Team  Hiring  The CV  The Technical Test  The Structured Interview  People Skills  Managing a Remote Team  Supporting the Team  Communication Channels  Listening  Politics  Answer Questions Before They're Asked  Daily \"Stand Ups\"  Building the Product  Understanding the Company  The Architecture  Code Reviews  Our Values        We say what we do and do what we say.    Note: this is a long article because this is a complex topic. Assembling and managing remote teams is largely like assembling and managing local teams, but there are a few differences. Further, though hiring is pretty much the same, since most people have little formal background in how to conduct a proper hiring process, I go in-depth in how to do this in a solid fashion which will gain you the best employees. This will largely be explained via our process of consulting, but it's appropriate for internal use, too.   I was living in Paris a few years ago and I was contacted by an old friend living in Brighton, England. Despite exhaustive searching, his company couldn't find any local developers experienced with Perl, nor could they find any developers both willing to learn Perl and work on their old, creaky codebase. After a bit of discussion, it was pretty clear what the issues were and what needed to be done to fix them. I've done this enough that I'm practically genetically engineered for this role and I was contacted because of my deep experience in this area.   So when I received the news that they hired a junior programmer with no Perl experience and who failed their language-agnostic programming test, I was a bit surprised. He did, however, have a key qualification I lacked.    I do not want to work here. Source    He was willing to sit in a chair in their offices for eight hours a day. I was telecommuting. Oh, and he was probably cheaper.    The situation went about as well as you would suspect and my friend eventually left the company when it was clear things weren't getting better. Apparently their junior developer couldn't redesign a decades-old codebase in a language he didn't know.  So why would a company make such a silly mistake? Fear . Developers often want to rewrite instead of refactor simply because they don't know how to rebuild a large code base. Similarly, management often says \"no\" to remote teams because they don't know how to manage one.  And I get it. Managing a remote team requires a subtly different skillset than managing a local team, especially for managing communication. However, once you learn how to build and manage a remote team, you have your pick of of some of the best talent on the planet . Think about having a team of top experts at your fingertips to build real value for your company. This is the real strength of remote teams and this is how you manage them.  Building a Remote Team  You can have the greatest management skills on the planet, but if you have an awful team, you're going to struggle.  Hiring well is the most important thing a company can do.  Every bad hire is flushing money down the toilet. Every bad hire takes the place of the person you really wanted. Every bad hire distracts other employees as they have to pick up the slack. Every bad hire tells other employees \"management doesn't understand.\"  And so it's worth repeating: hiring well is the most important thing a company can do.  That said, we have to identify what you're looking for. What you want in a remote team is what you want in a local team: a team that can respond to your needs and cares about success. In fact, most of what I say here will apply to local teams. Our consulting firm, All Around the World  , has a complete hiring process that focuses on both hard (technical) and soft (interpersonal) skills.  Hiring    Appropriate Working Conditions Source    Our hiring process is in three phases:   CV (résumé) review  Technical test  Structured interview   The CV  Many would-be employees get upset that HR departments often use software to filter based on keywords and phrases, but after years of hiring, sometimes in recruiting for other companies, I understand why HR firms do that. We don't use such software, instead choosing to hand-review every CV. But honestly, CVs are often worthless. First, many people lie on their CVs  . In fact, as I've seen on a number of reports, this number appears to be increasing in the past few years.   Other CVs are terribly dull, but have great minds behind them.  Thus, many experienced HR people will tell you: just use the CV to screen out the obviously unqualified. You need someone with a Java+Hibernate background? Ignore CVs that don't have Java on them. You can probably ignore CVs that don't have Hibernate on them. Make sure your CV lists the core hard skills you really need and if that's not on the CV, ignore it. It's amazing how many CVs we filter this way.  Also looks for duration of their experience. If you need top talent, don't go with someone who used your core technology on a three-month side project.  And because CVs are almost useless, that's pretty much all I have to say.  The Technical Test    Don't stress! Source    After we filter the CVs, we then offer candidates a chance to take our technical test. A technical test must not be Trivial Pursuit! Yes, I happen to know what the $\" and $^V variables are for in Perl. I can never remember what the $&lt; or $&gt; variables are for. That's OK, because if trivia is what you're asking about, you'll get trivial employees. Instead, you're looking for people who can think clearly about your problem domain. For example, when we're hiring backend software developers, our technical test is a small program we ask them to write, using a CSV file we provide. It might look roughly like this:  Write web-based software that allows someone to upload the included CSV file and saves the data into an SQLite database. Assume more CSV files in a similar format will be uploaded later. Create reports that show ...  There are a few other things, but we give the candidates a week to complete the test and send us the results. To make sure the test is fair, I take it myself. The above version of the test took me about two hours to complete the first draft. Again, candidates get a week to do this and we don't send them the test until they say they have the time to complete it.  And here's our secret sauce: we tell the candidate that they should return something they consider \"production ready\", but we don't tell them what \"production ready\" means. Thus, instead of writing to meet our needs, they tend to show us what they think \"production ready\" means.  For us, we're looking for well-structured code, with tests and documentation. We're looking for a properly normalized database   . We're going to see if you noticed the data quality issues in the CSV file. We're going to see if your code is vulnerable to injection attacks. We're going to see if you've used appropriate modules. In short, we check many things.   and we use a checklist to track our findings. We usually won't fail you for missing just a couple of items, but very few developers pass this test.  And in return? The developers get a full code review, including references of where they can read more information about those areas on which they were weak. I've been amazed at how many developers we turn down who thank us for that code review. Many of them have never had one before!  The Structured Interview    Not how we conduct interviews. Source    The final part of our individual hiring process is the structured interview. Very few people make it this far and this is the part where most hiring processes break down. Most interviews are worthless.  From the article:  Countless studies show that the unstructured 30-minute interview is virtually worthless as a predictor of long-term performance by any criteria that have been examined. You have only slightly more chance of choosing the better of two employees after a half-hour interview as you would by flipping a coin.  You may as well just chuck half your CVs in the bin and say \"I don't hire unlucky people.\"  The alternative is the structured interview. Here's a choice quote from a meta-analsis of 200 articles and books spanning 80 years  :  In the 80-year history of published research on employment interviewing (dating back to Scott, 1915), few conclusions have been more widely supported than the idea that structuring the interview enhances reliability and validity.  The analysis is quite in-depth, but it becomes clear that we go from the unstructured interview&mdash;well-proven to be almost worthless&mdash;to the structured interview&mdash;a powerful hiring tool that does an excellent job of helping you select the candidates who will actually be a great fit.  They actually take a fair amount of time and effort to create, conduct, and evaluate, but here's the short version.  First, the interview is primarily about assessing soft skills which are harder to teach, though we do throw in some technical questions to cover potential gaps in the technical test, based on particular client needs. So you take stock of what you really need for a candidate to succeed. Our company, All Around The World  starts by interviewing the client to find out their work environment. Are there tight deadlines? Do you need to work directly with customers? Do you have to communicate with non-technical stakeholders? Have you worked with remote developers before? And so on ...  From there, we assemble a list of open-ended questions that help us to better understand how the candidate's background fits. For example, in a fast-paced environment, the candidate might find themselves dealing with tight deadlines, so we might ask a question like \"tell us about a time when you had a tight deadline and you were concerned you might not be able to deliver on time.\" And then you apply STARR.   S ituation  Describe the problem you faced.  T ask  Describe your role in addressing the problem.  A ction  What did you do to address the situation?  R esult  What was the end result?  R eflection  Is there anything you would have done differently?   By asking every candidate the same question in the same order, and ensuring you have covered all of the points of STARR for every question (and taking careful notes the entire time!), we get a thorough and relatively unbiased view of the candidate's abilities in relation to the client's needs.  You want to have at least two interviewers for every candidate and immediately after you're done with the interview, you score the candidate (we use 1 to 5) for how well they meet the client's needs for each question. And then you just add up the points.   It works amazing well and produces stellar candidates.  A few points about structured interviews are worth noting. First, they're not often used simply because most people have never heard of them. Second, as previously mentioned, they're a time-consuming hiring process, but it's worth taking more time up front to avoid getting a \"dud\" employee. Third, candidates sometimes find it stressful to have you drilling into each question in such extreme detail, so be aware of that.    People Skills  Please note that throughout the entire hiring process, you're looking for people with good people skills. A destructive personality wrecks teams. I don't care how good they are, if they're a jerk, they ruin things for everyone. They need to communicate easily, be friendly, challenge without arguing, and generally be pleasant to be around. Don't hire toxic employees.   Managing a Remote Team    Probably not one of our managers. Source    Now that you know how we hire individuals, how do you create a remote team? This is where really knowing what needs to be done shines. You need people with different, but complementary skills. Some people might be brilliant at front-end while others are brilliant at backend. You might have someone who can see all possible complications and design a system which can handle all of them, but have another person who's good at simplifying designs and applying the \"YAGNI\" (You Aren't Gonna Need It) principle ruthlessly. Another person might be very skilled at building tools which just make life easier for other developers (we like a devops person embedded in the teams directly). By having a wide variety of skills, you get more well-rounded solutions and higher quality work.  These are people who can put their ego aside and have the willingness to work together to provide the best solution. And because they have diverse skill sets, they learn to trust one another when someone else is addressing an area of their expertise.  Important: creating a team with diverse skills is even more important for a remote team. Just because you client says they have a DBA or a DevOps engineer doesn't mean those people will have time for a new team. They have other things to do and they may not prioritize you as much as the friends they work with and know.  Supporting the Team  The absolute number one obstacle (and objection) to remote teams is communication. Sometimes that quick email someone dashed off misses key details about a particular task that needs to be done. Sometimes the employee gets finished waiting on a task and is asking around for work to do ( but see my \"Project Management in Three Numbers\" article for a great way to work around that ). And you can't just pop your head over the cubicle wall and say \"hey, Sally, where in the code do we handle authorization?\". So your role as a remote manager is to ensure that everyone always has the information they need.  This need for extra communication and working very hard on this communication seems challenging, but we've found that this makes our teams often more effective than local teams because sometimes local teams make assumptions about how things are, when our team can't afford to do that.  Communication Channels  One of the beautiful things about remote work is that you skip the office chatter, the constant interruptions, the need to beg IT services to let you use a different editor, and so on. I often work with my terminal fullscreen so I can engage in deep work  on heavily demanding tasks. But when I need to talk to someone about something, I need to have quick and easy to use communication channels. And email isn't one of them.  The tools you'll adopt will be different depending on your needs, but IRC, Slack, Zoom, Google Hangouts, and other tools will all be part of that arsenal. Whatever you do, ensure that you have at least one visual means of communication (but don't forget accessibility requirements your team members may have). It's great to ask a quick question on Slack about which hashing algorithm you should use, but nothing beats a quick Google Hangout session for quickly discussing complicated topics.  Listening    Listen and listen hard . Source    Agile fans like the term servant leader  because it makes it clear that management is there to support the team getting things done. Managing remote teams reinforces this. When your team needs you, they need you. Yes, you're setting goals, providing direction, and often serve as the voice of the business, but when Oliver says \"I can't get QA to test my branch,\" then you follow up and make it happen because Oliver's job isn't fighting with QA. When the team says there's a problem, your job is to clearly identify what's wrong and get it sorted . This is much like the ScrumMaster role of removing any and all blocks for your team.  Sometimes management can seem distant or not paying attention—they often are juggling multiple things at once and this can be distracting. Managing remote teams cannot be like this. Especially when you have your daily \"stand up\" (scare quote explained later), you must be focused and care what your team is doing.  Politics  One issue I've seen about remote teams is that they have a particular sensitivity to office politics. This may seem counter-intuitive because your team members aren't standing around the office coffee machine, complaining that Martin isn't doing his job; they're working. The problem is that they're often anonymous to other teams.   You need to build bridges to the other teams and bridges aren't supported at only one end.    Note how the bridge is cleverly supported on both ends. Source    When different groups have different priorities and there are limited resources (as there always are), it's very easy for local teams to have the \"we were here first\" attitude. Or maybe your remote team needs a new database provisioned, but the people responsible for it don't know who you are and, more importantly, have other things they need to do.  Your job is to fix this, but it's a delicate task. Yes, you can often go to senior management and say the work you've contracted to do is stalled because other teams aren't cooperating. And, especially if you're on a critical project, senior management will sometimes respond by stepping on the foreheads of those who aren't helping you.  This exchanges a short-term problem for a long-term one.  Senior management doesn't want to step in every time you're struggling and this can leave the impression that you can't solve things for yourself.   As anonymity is a classic driver of politics—you're not human to them until you meet them—fix that and other problems will be much easier.  Get to know the other teams and managers, paying special attention to those you'll be working with. When you have stand-ups, invite representatives of other teams to your Zoom meeting. If they have remote meetings, offer to be present, explaining you'd like to learn more, or offer help if need be. When they need your support, give it to them, too. Are they struggling to figure out why their PostgreSQL database is running slowly and you happen to have a PostgreSQL guru on hand? Offer that support if you can. People don't want to help if they view it as a one-way street.  You are a helpful, friendly, rising tide that lifts all boats. Helping one another and making sure everyone knows one another helps to minimize politics quite a bit. But they'll still be there. No matter how important your work, the work you're not doing is important too. Some other manager might think you're the greatest thing since sliced bread, but she has a tight deadline and no, the bug your team identified won't get fixed by the next release.  That's a realistic position, but it gets called \"politics.\" When you let your team know the bug can't be fixed right now, you're supportive of your team, but of your client, too. If the team insists on the full story, tell them the truth, but tell it from the standpoint of the manager who denied the request. \"She would love to help, but she doesn't have the resources to fix this before the next release.\"  By doing this, you show support for everyone . You're like Tom Hanks as Captain John H. Miller in \"Saving Private Ryan\": you don't grumble or complain. You support. You build bridges to others and you shield your team from the mess. They will learn to trust you, they'll reserve their complaints for critical issues, and they'll have fewer distractions from their work.  Answer Questions Before They're Asked    \"Psst! I have a question!\" Source    This is one of the hardest skills to develop, but it goes a long way to ensuring you have a great team. Often your team will say things like \"we tried to extract the data but we can't because the API appears to be inconsistent.\" If your answer is \"I'll find out immediately what's going on\", that's great, but it means your team is blocked until you come back with more information.  A better answer is to realize they're dealing with an external resource and find out who's an expert you can call on right away. Then your answer is \"Damien knows the best about that system and he can provide you with documentation and answer your questions.\"  Curiously, non-technical managers are sometimes better at this. A technical manager might have some experience configuring SNMP for Cisco routers and in a meeting, might not realize that SNMPv2 is insecure and SNMPv3 is more secure. The manager may not know anything about the NETCONF alternative and be unaware that the suggested polling solution doesn't scale with SNMP.  Thus, a technical manager might gloss over details, thinking they understand (this is not a given, of course), while the experienced non-technical manager learns to swallow their pride and ask why there's a mixture of SNMPv2, SNMPv3, and NETCONF solutions being used. Your job is to listen and listen hard and if you must leave a meeting with unanswered questions, you have a list of them and will immediately start tracking them down. Don't make assumptions that \"they'll figure this out\" because as a manager, your job is to ensure that you have the answers and the information flows freely.  This also means not shying away from the pain of a late-night session reading through documentation and checking it against your notes to ensure that you haven't made bad assumptions. You're not providing solutions, your team is, but you are there to guide them and you can't if you're not able to address their questions immediately.  Remember: one of the greatest obstacles facing remote teams is the communication flow.  Daily \"Stand Ups\"  The daily stand-up is our biggest change from traditional management techniques, particularly agile ones.  The name \"stand-up\" comes from the idea that if you're all standing up and some team members are starting to lean, the conversation has gone on too long. Thus, stand-ups are often time-boxed to 15 minutes. This does not work as well with remote teams.  The idea of a standup is to have every team member answer three short questions:   What did I do yesterday?  What will I do today?  What blockers do I have?   We've found that the 15 minute time-box doesn't work as well with our \"complete communication\" style. The reason is simple. When everyone's in the same room and someone's explaining a technical problem at length, you often hear the ScrumMaster say \"let's take that offline\" because it's disrupting the flow of the meeting.   Except with remote teams, there's no \"offline.\" You can't turn to the person next to you and say \"hey, let's talk about that problem I had.\" Even if you do, you're not fully sharing the information because not everyone hears about it.    Working on code you don't understand. Source    And if someone identifies a blocker, you address it in the stand-up , you don't \"take it offline.\" Everyone on your team needs to know you're there for them if they're waiting on a code review, if they can't figure out what MyCompany::Catalyst::Role::RewritePaths does, or they have a test failure they can't figure out. Ask for solutions, discuss the issue, have someone volunteer to look at their code. And if you must, assign someone to review their pull request. Someone might grumble that they're being taken away for a code review, but they know their code will be reviewed in a timely fashion too.  We've found that our stand-ups tend to run about 45 minutes to an hour. Every day. That takes more time, but when you have a good team, they absorb more information. Even if they don't remember all of the details about why you changed your connection pooling software, when it's discussed more in-depth, they're more likely to remember that it was done and know who to talk to when they encounter difficulties with it.  We also use stand-ups as a time for chatting and socializing. This is not the norm, but it's easy to feel isolated when working remotely and this gives the team a chance to bond, to share stories, and laugh. All work and no play ... well, you know the rest. Being part of a team of friends, and not just co-workers, is important. When I go to conferences, our employees and contractors are often there and we take them out to dinner and share war stories. It's great to build loyalty, but more importantly, it's the right thing to do.  Note that this suggests a change in your hiring strategy. If candidates on your Google Hangout are getting fidgety towards the end of a long interview, that might be a warning sign (or maybe they have another appointment). If you mention your stand-up approach to candidates and they immediately react with \"stand-ups should not last more than fifteen minutes\", that's another warning sign that agile has become dogma to them rather than a set of guidelines. You're looking for candidates who are willing to communicate well. That means both being able to explain themselves clearly and to listen attentively.  Building the Product  Just taking a work specification and working through the bullet points is a common recipe for disaster for remote teams. Instead, you have to build value; the software just comes with the territory.  Understanding the Company  When we're brought in to a company to help with an issue and have hired our team, we work hard to understand the business of the company and the problem they're facing. And then there are other concerns which might impact our solution. Do we have to address PCI-compliance? Is the company impacted by GDPR? Does the company have restrictions on the types of software they allow? And what's the full stack they have available and are we allowed to offer alternatives, if necessary? Building a real-time reporting system with heavy writes using a MySQL database using MyISAM tables requires a substantially different architecture than Oracle 12c.  Once we have that information and can share it with the team, the hard work begins.  The Architecture  As I've written before , the later in a project that a bug is found, the more expensive it is to fix. It's easy to fix a leaky sink; it's disastrous to realize your foundation can't support the weight of your house.    The architecture of hard problems. Source    So we start with a careful evaluation of the problem, the kinds of data we're working with, and the resources we have at our disposal. Then we discuss possible solutions to the problem, with the entire team and their diverse skills contributing in their various areas of expertise. If possible, it's great to have a senior person from the company on hand who has deep experience with the codebase and company and can offer invaluable advice in driving the architecture.  If the problem is small, this might be a short meeting. If it's large, this might be a series of meetings. Regardless of the approach, the client must be able to review the architecture and offer appropriate feedback before signing off.  After hiring well, a strong architecture and review is the best thing you can do to reduce bugs and control costs.  Note that unlike code reviews, architecture reviews happen before implementation.  Code Reviews  After the architecture is approved, the implementation begins. How the work is organized depends on your management style, though I confess that I'm a fan of WildAgile and the three-number business case . Once a team member has a task, they might discuss it briefly to clarify the requirements, and when they're done working on it, they submit it for a code review.  After hiring and architecture, this is the third most important thing in assuring code quality. Even if your code works perfectly and your tests are passing, tests are not enough. Tests generally don't know when a public method should be private. Tests often can't point out that you forgot to test edge cases or that you missed a race condition. And tests certainly can't tell you that your technically correct code and documentation is nonetheless incomprehensible.  Thus, code reviews. When we do code reviews, we're checking that the code is correct, that it's understandable, that it's documented and that it has appropriate tests. More than half of all of our pull requests are sent back for more work, but that's normal for a team that is heavily focused on quality. The larger the pull request, the more likely it is to have some issues.  The combination of a thorough understanding of the problem domain, careful work on architecture, and deep code reviews means that when issues do arise in the software, or a change of requirements occurs, there are seldom systemic issues. This reduces the up-front cost of development and tremendously reduces maintenance costs.  Our Values     We say what we do and do what we say.    When working remotely, it's essential that the people you are working with share the same values. We work hard to find people who excel, are proud of their work, but can put aside their ego and focus on the client's needs.  For us, the value is to be able to help, to take the client to another level. We are putting our talent at the service of the client to support their growth. We are not there for billable hours. We are there for delivering; for supporting.  Whether you choose us or choose to develop your own remote team, following these guidelines will give you a superb team that can do things with software you never had the chance to do before.  ","title":"Building Remote Teams","url":"\/articles\/managing-a-remote-team.html"},{"body":" Summary  If you have EU customers and you're uncertain how GDPR impacts you, stop reading this and hire a lawyer. Now.  Disclaimer  This is not legal advice. If you have EU-resident customers, hire a lawyer. Now.  Background  I work for All Around The World , a consulting firm based in France. Not only have we been impacted by the General Data Protection Regulation (GDPR) law, our clients have as well. What we have seen is that there's often a lack of understanding of the consequences of GDPR and companies with EU customers face bankruptcy if they fall afoul of this legislation. The GDPR essentially forces companies to take security, privacy, and record keeping seriously. GDPR fines for non-compliance are designed to be severely punitive, up to and including driving companies into bankruptcy.  You do not need to suffer a data breach to be hit with these fines; if you regularly handle data on EU customers and you're found to not be complying with GDPR regulations, you may be sanctioned. What's worse is that local laws, such as the US CLOUD Act, are sometimes in conflict with the GDPR , meaning that you can be caught between contradictory legal requirements. You need a lawyer.  The EU worked for years on the law, finally adopting it on April 14th, 2016. It become enforceable on May 25, 2018. Companies had two years to get ready, so non-compliance is not tolerated. The Cambridge Analytica scandal has made the situation even worse. Though Facebook was fined £500,000 for misusing customer data , there's anger that this paltry amount was the maximum fine that could be levied at the time. Had GDPR been in effect at the time, Facebook could have been fined up to $1.8 billion . While \"good faith\" efforts to comply with the law will be taken into account, if the EU desires to make an example of a company (and they will), a US company taking money out of the EU will be a much more desirable target than bankrupting an EU company keeping money in the EU. You need a lawyer.  The Legal Situation  There is a strict requirement that you make a \"good faith\" effort to achieve GDPR compliance. Before anything else, you must realize that GDPR-compliance is a legal issue, not a technical one. If you simply instruct your IT staff to make it easy to report and purge personal data, you're well on your way to bankruptcy. This is not a good faith effort because it demonstrates a lack of knowledge of GDPR regulations.  If you are found to be non-compliant, there are two levels of fines :   Up to the greater of 4% of global annual turnover or €20 million for the most serious failures, such as insufficient customer consent.  Up to the greater of 2% of global annual turnover or €10 million for the other failures, such as not having a Data Protection Office (if required), failure to report breaches, or not conducting an impact assessment.   In other words, you face bankruptcy if you get this wrong. This is serious enough that many US-based Web sites are no longer available here in Europe . Complying with GDPR can be costly if your IT staff are inexperienced. Not to put too fine a point on it: the strong worldwide demand for software developers has led many companies to hire inexperienced developers. Even expert developers often build poor systems when faced with significant budget or time constraints. Companies rush build software to solve problems now with little thought to the long-term consequences of poorly-built software . GDPR isn't going to improve this situation, but it's going to severely punish companies who get this wrong.  Basic GDPR Compliance  In order to achieve GDPR compliance, you need to, at minimum, do the following, keeping in mind that there are major caveats to each of these:   Request consent from your customers to track their data.  Appoint a Data Protection Officer, if required.  Create a Data Protection Impact Assessment (DPIA).  Maintain records of all personal data processing activities.  Notify EU authories of data breaches with 72 hours.  Understand the \"right to be forgotten.\"  Provide customers with information on how you use their data.   How these issues are applied will vary company to company. However, keep in mind that you generally do not have the right to deny services to customers if they fail to give consent to having their data tracked . There are exceptions to consent, such as the processing of personal data for reasons of public interest, such as in the health sector . And yes, this also applies to your EU-resident employees, not just customers.  Data Protection Officers  In short: if you handle personal information on a large scale or regularly monitor data subjects, you need a Data Protection Officer (DPO) . Keep in mind that Recital 30 of the GDPR clarifies that IP addresses are personal information. Examples of companies requiring a DPO include:   Any government body (except courts) or public authority  E-commerce companies tracking customer purchases  Hospitals and clinics  Advertising firms targeting personal data  Telephone or internet providers  Virtually any company offering financial or insurance services  Geo-location services used for statistical purposes  Recruiters who store candidate information  Security firms monitoring public spaces   Small scale tracking of personal data, such as a individual doctors or lawyers, generally do not need a DPO.  Of course, the above list is not exhaustive. If you're not sure, consult a lawyer.  If you require a DPO, the following conditions apply :   They must not have a conflict of interesting (controllers, IT directors, CEOs, etc.)  They must have sufficient budget and personnel to perform their task  They report directly to top management and must not have a supervisor  They may be an outside consultant or firm, but they must not have a short or fixed-term contract  Their term must be between two to five years, but may be extended up to ten years  They must be knowledgeable in both GDPR-compliance and your internal systems (it's understood that the latter will take time)  To protect against retaliatory dismissal, they can only be dismissed with the consent of the European Data Protection Supervisor (EDPS)  They must have full power to investigate and correct GDPR compliance within the organization  They must notify the EDPS of any data processing activity that is likely to present significant risk \"to the rights and freedoms of data subjects\".   The above list, of course, is not exhaustive.  IT Considerations  Now, and only now, do we begin to touch on the IT considerations. The above was merely to give you a sense of the scale of the GDPR directives. This section is intentionally left short because it should not be viewed as a checklist to becoming GDPR-compliant. If in doubt, consult a lawyer.  The legal implications must be addressed first. One you have a DPO (if needed), and have drafted a comprehensive plan to protect your customer's data, you can start work on how to implement this. Implementing GDPR requirements without fully understanding them risks wasting money and time developing systems that are not fit for purpose. Once you have a strong understanding, however, you can begin to address the following:   Asking for consent in a clear, intelligible manner that gives consumer full control over how their data is managed  Develop reporting to track usage of all personal data  Respond to consumer requests for your usage of their personal information  Respond to consumer requests for the \"right to be forgotten\"  Ensure that disaster recovery and data backups do not restore \"forgotten\" information  Restrict internal access to sensitive data  Hire a reputable, external company to do a security audit and have developers fix discovered issues   The security assessment is critical: all the \"good faith\" in the world isn't going to protect you if you have a Experian-style data breach . The EU wants to show that the GDPR has real teeth and you don't want to be the example.  ","title":"GDPR and Bankruptcy","url":"\/articles\/gdpr-and-bankruptcy.html"},{"body":" This article is longer than most. That's because it's more important than most. If you avoid these software mistakes when starting a new project, you will save yourself a lot of grief. And by grief, I mean \"money.\"  My new client was playing the long game. They needed data for analysis. Lots and lots of data. They had been saving customer data for two years, waiting for the critical mass at which they could have their data science team dive into it. I wasn't on this project, but I was doing some work on their system when I happened to notice some warnings from their database that I was all too familiar with: Data truncated for column ... .  No, that wasn't an error; it was a warning. Many of you are familiar with that horror story: MySQL wasn't being run in STRICT or TRADITIONAL mode. . Further investigation revealed that the data being truncated was the key the client was using to link this valuable customer data together. It was impossible to reconstruct. Two years of valuable data had to be thrown away because the client didn't know how to configure MySQL.  This, and many other disasters, can be avoided by taking a little time up front to understand common errors on large projects.  The Database  I find it interesting that when I go into a shop running Oracle or MySQL, the database is usually a mess and there's lot of work to fix there. When I go into a shop running PostgreSQL, the database is usually fine. I can't prove this, but I suspect it's because people who really understand databases reach for PostgreSQL due to its obsession with data quality, while people who just use databases reach for MySQL because it's \"easy\" or Oracle because \"the company said so.\"  As a further data point, when I hire people, there's a small technical test. I've given that test hundreds of times. It was three years before a single developer returned that test with a properly normalized database. There were no \"gotchas\" and the database itself only required five tables to be normalized correctly. Many of these were excellent developers with a decade or more of experience but nonetheless turned in sloppy databases. It was trivial to demonstrate potential data anomalies in their database . Most software developers don't understand databases.  By running a default configuration of MySQL, you can get invalid dates, truncated data, NULL values for division by zero instead of an error, and so on. MySQL and MariaDB have other issues, but this is the biggest one. Every one of my clients who has used MySQL and tried to get themselves into STRICT mode have found their software assumes sloppiness in the data and can't run with a properly configured database.  Recommendation: if you must use MySQL, ensure you're using one of the recommended modes and have a competent DBA who understands database design and performance. Otherwise, if there's not strong requirement for choice of database, use PostgreSQL. I've heard DBAs complain that it takes more work to administer, but that's because motorcyles require more maintenance than bicycles, but they'll get you there faster.  Database Migrations  When dealing with new clients:  Me: \"How do you update the database?\" Client: \"Run this script. It searches through the SQL directory to find the latest SQL we checked in. It runs that in your database.\" Me: \"How do you back out the change if there's a problem?\" Client: \"Uh, there's hardly ever a problem, but we fix it by hand.\"  And things go downhill from there. I frequently meet clients with insane database migration strategies. The \"dump SQL in a directory\" for people to apply is an annoyingly common strategy. There's no clear way to roll it back and, if you're using MySQL or Oracle, if it contains DDL changes, those aren't transaction safe, so they really should be in their own migration, but they're not.  A few clients have an in-house database migration strategy involving numbered migrations. It often looks like this:  ... 213-up-add-index-on-address-state.sql 213-down-add-index-on-address-state.sql 214-up-add-customer-notes-table.sql 214-down-add-customer-notes-table.sql 215-up-add-sales-tax-sproc.sql 215-down-add-sales-tax-sproc.sql  That, at least, can allow devs to back out changes (but tricky if your DDL isn't transaction-safe), but it's amazing when you get to migration 215 and you have 30 developers, five of them need to make a datababase change an they're arguing over who gets number 216. Yes, I've seen this happen more than once.  With a naïve numbering strategy, you can't declare dependencies, you get numbering conflicts, you really can't \"tag\" a particular migration for deployment, and so on.  Or there are the migration strategies which require migrations to be written in a particular programming language. Those are often nice, but can't always leverage the strength of the database, often write very poor SQL, and make it hard for other teams not using the language to write migrations.  Recommendation : just use sqitch . It's well-tested, not tied to a particular framework or programming language, let's you write changes in native SQL, declare dependencies, tag changes, and so on.  Oh, and it's free and open source.  Time Zones  This one still surprises me, but many clients are storing datetimes in their database in \"local time.\" Most of the time this means \"not UTC.\" I'll just get the recommendation out of the way now.  Recommendation : whenever possible, store dates and times in the database in UTC.  Many shops started out small and many datetime tools report the local time because if you're in Portland, Oregon, you get confused when it's 1:30 PM but your time function says 8:30 or 9:30 PM.  But it's worse than confusion. I've seen many test suites fail twice a year ... right when DST starts and ends. That costs money!  Or here are some fun problems documented in the Perl DateTime module: Ambiguous and invalid times.  Ambiguous Local Times  Because of Daylight Saving Time, it is possible to specify a local time that is ambiguous. For example, in the US in 2003, the transition from to saving to standard time occurred on October 26, at 02:00:00 local time. The local clock changed from 01:59:59 (saving time) to 01:00:00 (standard time). This means that the hour from 01:00:00 through 01:59:59 actually occurs twice, though the UTC time continues to move forward.   Invalid Local Times  Another problem introduced by Daylight Saving Time is that certain local times just do not exist. For example, in the US in 2003, the transition from standard to saving time occurred on April 6, at the change to 2:00:00 local time. The local clock changes from 01:59:59 (standard time) to 03:00:00 (saving time). This means that there is no 02:00:00 through 02:59:59 on April 6!  Using UTC doesn't make all of your problems go away, but because UTC doesn't use DST, you'll never have invalid or ambiguous times.  Time zone issues seem rather abstract, but as your company grows, these problems become more severe. In particular, if you work with other companies, they will be very unhappy if you cut off their services an hour or two early simply because you got your time zones wrong. Or in one memorable case (NDA, sorry), a company printed up a bunch of schedules with all times an hour off because of this.  One of my clients in California tried to switch to UTC because they had gone worldwide and local time was causing them headaches. When they realized how much time and money it was going to cost, they switched to Mountain Standard Time (Arizona) because they don't use daylight savings time. My client simply accepted that all of their old dates would forever be an hour or two off.  NoSQL  One of my favorite interactions with a developer was in the Paris office of a client with legitimate \"big data\" needs. But their NoSQL solution (which I'm not going to name) was awful. It was slow, painful to query, but offered \"eventual constistency.\" There was a different NoSQL solution that seemed like a better fit, including a much better query syntax. However, if you learn one thing as a consultant, let it be this: never assume your client is stupid. If you're wrong, your client gets angry. If you're right, your client gets angry.  I approached a senior dev who was there when the NoSQL decision was made, laid out my findings, and asked \"what am I missing? I need help understanding your NoSQL choice.\"  He replied, \"we misread the documentation.\"  I laughed; he didn't. He wasn't joking. This multi-million dollar business made a very expensive mistake that they couldn't easily back out of because they misread the documentation.  This is not unusual.  Many developers, me amongst them, get excited about new technologies. Either we want to explore new things (that's why we became developers!) or we want to pad our CVs. Neither of these is necessarily in the company's best interest. While I sometimes recommend NoSQL (I love Redis, for example), usually I urge caution. I've seen client after client reach for NoSQL without understanding it. Here's what you need to understand:  Before SQL, everything was NoSQL. That's why we have SQL.  If you're considering a NoSQL solution, here's a handy checklist for you:   What problem are you trying to solve and do you really need to solve it?  Why does your database not solve that? (NoSQL is a premature optimization if you don't have a database yet)  How can you fix your database to solve that problem? (I'm amazed at how often this step is skipped)  If your current developers can't answer the previous question, have you considered hiring an expert? (Often far less expensive than introducing a new technology)  Why does a NoSQL solution solve this problem better?  What type of NoSQL solution do you really need? Key\/value? Column-oriented? Graph? And so on ...  What limitations does your NoSQL solution have and how are you compensating for these?   Often when I see clients switching from SQL to NoSQL, I find they're trading a known set of problems for an unknown set of problems. How is that good for your business? NoSQL has its place, but given how many developers are don't understand databases, why do we magically assume they're going to understand the byzantine NoSQL solutions available?  Internationalization  If there is any chance that your software will ever need support for multiple languages, investigate building that support in from day one. One of my clients decide to expand from the US to the UK. Easy, right? Hmm, date formats are different. Currency symbol is different. And the subtle spelling differences were \"labour\"-intensive to fix. It cost them so much money relative to their meager earnings abroad that they canceled the project. They might revisit it someday, but not now.  When looking at internationalization (i18n-getting your code ready for localization) and localization (l10n-the actual translation and other changes necessary), even if you're not going to launch in another language, make a \"secret\" translation feature. Heck, make it Pig Latin if you want. Even if you get it wrong, it won't really matter because it's not for public consumption. Instead, render everything in Pig Latin and check the software (better, write a spider that can find untranslated strings). One you get over the initial hurdle, it will become second nature to not hard-code strings or date formats everywhere.  Recommendation: Start your i18n work from day one. Trying to change it later is very expensive.  The Takeaway  You may have noticed a pattern in the above. Early mistakes are often hard to notice until they turn into expensive mistakes.    Source: IBM System Science Institute Relative Cost of Fixing Defects   The sooner you catch bugs, the less money they cost to fix. When you're dealing with well-known categories of bugs, you have little excuse for them. For open source databases, I can't imagine starting a new project with MySQL instead of PostgreSQL. Of course you need to store times in UTC. Just say \"no\" to NoSQL until you can prove you need it.  Of the problems I mention above, them can all be fixed in the design phase but are often caught in the maintenance phase where it's literally two orders of magnitude more expensive to fix: every dollar you didn't spend up front becomes $100 dollars later, and that's not counting the money you lost because you didn't fix it.  I'm a huge fan of agile development , but agile shouldn't mean checking your brain at the door and it definitely shouldn't mean \"don't think of the big picture.\" You'll save yourself a lot of money and grief if you take a little more time when you start a new project.  If you have any other \"up front\" design issues like those above, mention them in the comments below! Or better yet, tell me how many your project has made!  ","title":"Avoid Common Software Project Mistakes","url":"\/articles\/avoid-common-software-project-mistakes.html"},{"body":"  For those curious about the current status of the Corinna OOP project for Perl  , Paul Evans is working on it in the feature-class branch  . Today I decided to see how much of my Cache::LRU example I could get working.  Under the full proposal, Cache::LRU looks like this (it’s unclear if the :handles modifier will be implemented for the MVP):  class Cache::LRU { use Hash::Ordered; field $cache :handles(exists) { Hash::Ordered-&gt;new }; field $max_size :param :reader { 20 }; method set ( $key, $value ) { if ( $cache-&gt;exists($key) ) { $cache-&gt;delete($key); } elsif ( $cache-&gt;keys &gt;= $max_size ) { $cache-&gt;shift; } $cache-&gt;set( $key, $value ); # add to front } method get ($key) { if ( $cache-&gt;exists($key) ) { my $value = $cache-&gt;get($key); $self-&gt;set( $key, $value ); # add to front return $value; } return; } }  This is what I currently have working:  class Cache::LRU { use Hash::Ordered; # Delete arguments to constructor or else they'll cause a fatal # error when new() is called. When :param is added, this will # no longer be needed. field $max_size; ADJUST { $max_size = delete($_[0]-&gt;{max_size}) \/\/ 20 } field $cache; ADJUST { $cache = Hash::Ordered-&gt;new }; method max_size { $max_size } method set( $key, $value ) { if ( $cache-&gt;exists($key) ) { $cache-&gt;delete($key); } elsif ( $self-&gt;num_elements &gt;= $max_size ) { $cache-&gt;shift; } $cache-&gt;set( $key, $value ); # add to front } method get($key) { if ( $cache-&gt;exists($key) ) { my $value = $cache-&gt;get($key); $cache-&gt;set( $key, $value ); # add to front return $value; } return; } method num_elements() { return scalar $cache-&gt;keys; } method exists($key) { return $cache-&gt;exists($key); } }  A few things to note:   Default initializer blocks are not yet implemented  All unprocessed arguments in the constructor are fatal (delete them)  Everything else works quite nicely   And here are the tests for it:  my $cache = Cache::LRU-&gt;new; is $cache-&gt;max_size, 20, 'Value of $max_size set by ADJUST'; is $cache-&gt;num_elements, 0, '... and our cache starts out empty'; ok !defined $cache-&gt;get('foo'), 'We should not be able to fetch values we have not defined'; ok !$cache-&gt;exists('foo'), '... and our exists() method should confirm this'; ok $cache-&gt;set( foo =&gt; 42 ), 'We should be able to set cache values'; ok $cache-&gt;exists('foo'), '... and our exists() method should show the new key exists'; is $cache-&gt;get('foo'), 42, '... and get the value back'; is $cache-&gt;num_elements, 1, '... and we should have one element in our cache'; $cache = Cache::LRU-&gt;new( max_size =&gt; 2 ); is $cache-&gt;max_size, 2, 'We should be able to set the max_size with a constructor argument'; ok !defined $cache-&gt;get('foo'), 'We should not be able to fetch values we have not defined'; ok $cache-&gt;set( foo =&gt; 42 ), 'We should be able to set cache values'; is $cache-&gt;get('foo'), 42, '... and get the value back'; ok $cache-&gt;set( bar =&gt; 'asdf' ), 'We can set a different key'; is $cache-&gt;get('bar'), 'asdf', '... and get back a different value'; is $cache-&gt;get('foo'), 42, '... and get the value back'; is $cache-&gt;num_elements, $cache-&gt;max_size, '... and we see that our cache is now full'; ok $cache-&gt;set( new_element =&gt; 3 ), 'We can set a new item in our cache'; is $cache-&gt;num_elements, $cache-&gt;max_size, '... and we see that our cache is still full'; ok !$cache-&gt;exists('foo'), '... and older elements in our cache should be ejected';  The tests revealed an “off by one” error in my original code (caches would contain up to max_size + 1 elements), along with a syntax error (both now fixed), but as it stands, I think everyone should be pleased with the progress Paul is making. Exciting times for Perl!  ","title":"Current Corinna Status","url":"\/articles\/current-corinna-status.html"},{"body":"    The Proposal  A Little Background  What is Inheritance?  A Bank Account Example  Conclusion     The Proposal  There has been some discussion of whether or not the new Perl OO model, Corinna  , should support exposing field variables to subclasses:  class Parent { field $name :inheritable; } class Child :isa(Parent) { field $name :inherited; }  There are a few benefits cited there.   If field $name doesn’t exist in the parent, the child class throws a compile-time error.  If the child can access $name directly, it’s a performance win (no method call).  By being explicit about :inheritable and :inherited , we’re not exposing data by accident.   It seems like a win, but it’s not.  A Little Background  Most OO developers today use class-based OO, but JavaScript is popular enough that prototype-based OO is getting better known. Then there’s Dr. Alan Kay, the man who coined the term “object-oriented programming” five decades ago and is considered one of the fathers of the concept. For him, OOP is actually about:   Message passing  Isolation  Extreme late binding of everything   So that’s another way of looking at OOP. And by “isolation”, Kay actually said “local retention and protection and hiding of state-process.” He would not approve of exposing field variables because it’s explicitly exposing the state, not hiding it.    Yeah, well, you know, that’s just like your opinion, man.   So clearly there are different ideas about how OOP should be implemented , but Kay is focusing on how to do it safely . He has a maths and biology background and he thought about the billions of cells in our body which die every day, but we don’t. He wants that sort of robustness in object-oriented code. To him, the only truly successful demonstration of OOP is the internet. Tons of servers crash every day, but the internet does not.  In fact, before he got to this conception of OOP, his original description of OOP left out inheritance because it was so problematic. The way I like to describe it is the Person :isa(Invoice) problem. Even if you can get that to work, it simply doesn’t make sense and until AI becomes a hell of a lot better, the software has no way of knowing if what you’re doing makes sense.  What is Inheritance?  Per Wikipedia, inheritance  is:   In object-oriented programming, inheritance is the mechanism of basing an object or class upon another object (prototype-based inheritance) or class (class-based inheritance), retaining similar implementation.   Some languages, such as Perl, allow multiple inheritance (Corinna does not). Other languages, such as Java and Ruby, only allow single inheritance, but provide you with tools (interfaces and mixins, respectively) as a substitute. In fact, inheritance is widely viewed as so problematic that Go and some Visual Basic variants don’t provide inheritance at all!  What’s worse, I’ve found that many developers kind of assume that behavioral inheritance and subtype inheritance are the same thing, but they’re not. Behavioral inheritance (which is what Perl tends to use), merely uses syntax to say “hey, I’m borrowing my parent class’s behavior.” There are no guarantees. Subtype inheritance, however, ... wait? What’s a subtype?  In the Raku language  , here are two subtypes of Int : Odd and Even :  subset Even of Int where * %% 2; subset Odd of Int where !(* %% 2); my Odd $x = 4; # we get an exception here  You can use the Even and Odd subtypes anywhere you can use an Int and your program is still guaranteed to be correct, but with the added safety of knowing that those variables will always be even or odd if they’re set.  Here’s an important concept to remember (you’ll be tested on it later!): the parent type does not know, or care, about what subtypes will be created from it. You can subtype anything in Raku (or other OO languages) and while subtypes know about their parents, the reverse is not true.  So subtype inheritance, used by languages such as Eiffel or Beta, guarantees that you can use an instance of a subclass anywhere you can use an instance of a parent class and the program still works.  Of course, that’s what the Liskov Substitution Principle  is all about.  If you’re subclassing, you probably want to enforce subtype subclasses, but neither current Perl OO (nor Corinna, to be fair), can do that easily (when\/if we get types\/type constraints in Perl, it will help).  So that gives you an idea about divergent views on inheritance and was probably boring as heck. So forget about that for now and let’s move on to an example.  A Bank Account Example  You need to create a simple, in-memory Bank::Account class. Obviously, this is a toy example, but bear with me.   You instantiate it with a customer name  Initial balance is zero  You can withdraw money, but throw an exception if the balance would go below zero  You can deposit unlimited amounts of money   Because Corinna, does not (yet) support type constraints, we’ll ignore them for now.  class Bank::Account { # Of *course* this is too simplistic. The first pedant # who points this out loses 500 internet points. use My::Exceptions qw(InsufficientFunds) field $customer :param :reader; field $balance :reader {0}; method deposit ($amount) { $balance += $amount; } method withdraw ($amount) { if ( ( $balance - $amount ) &lt; 0 ) { InsufficientFunds-&gt;throw(&quot;Naughty, naughty&quot;); } $balance -= $amount; } }  OK, so far, so good. The code works fine and everyone is pleased. Later, business requirements are updated because Mr. Moneybags has tons of money, but sometimes he wants to be overdrawn a bit. Since Mr. Moneybags is an important customer, you need to create a Bank::Account::Premium class to allow premium customers to be overdrawn.  You’re a good programmer, you know you shouldn’t rewrite code, so you just want to subclass it:  class Bank::Account::Premium :isa(Bank::Account) { sub withdraw($amount) { ... } }  But what goes in the withdraw method? You properly encapsulated your first class, so now you can’t get around that. In your Bank::Account class, you make a tiny change:  field $balance :inheritable :reader {0};  class Bank::Account::Premium :isa(Bank::Account) { field $balance :inherited; sub withdraw($amount) { $balance -= $amount; } }  And now, everything works fine and you move your code into production.  You them immediately move it back out of production because production is throwing exceptions left and right. Why? Your system knows that if $instance isa Bank::Account holds true, the balance can never be less than zero and because the code knows it can trust your Bank::Account class, it’s safe.  Bank::Account::Premium is a behavioral subclass, not a subtype subclass because the behavior of the child differs from that of the parent. It allows negative balances and the parent does not.  Oh, but maybe that doesn’t happen. I’m just being paranoid, right? Or maybe it only happens in one place and you can write one tiny little hack rather than remove this valuable functionality.  Months pass and your code is running fine and you’ve moved on to another department, when you get a frantic call to find out why everything is failing. After investigation, you discover the bug. Someone’s released a module allowing :isa type constraints and the person maintaining the Bank::Account module has realized that it’s safer to constrain the $balance :  field $balance :inheritable :reader :isa(ZeroOrPositiveNum) {0};  But your code which does allow negative balances blows up because it’s inherited this field.  Remember: the parent classes don’t know how they’re going to be subclassed and they shouldn’t care. Instead, they should do everything in their power to be as correct as possible and the :isa(ZeroOrPositiveNum) constraint is perfectly appropriate.  By exposing our internal state to our subclass, we’ve tightly coupled them because the cheap :inheritable workaround seems so easy . But again, Kay reminds us of “local retention and protection and hiding of state-process.” State and process are tightly coupled inside of a class and exposing state without the process that guarantees the state is correct leads to fragile code.  Here’s the real issue: ignoring encapsulation because it’s “easy” means we don’t have to think about our object design. Instead, we should have had something like this:  class Bank::Account :abstract { field $customer :param :reader; field $balance :reader {0}; method deposit ($amount) { $balance += $amount; } method withdraw ($amount) { $balance -= $amount; } } class Bank::Account::Regular :isa(Bank::Account) { use My::Exceptions qw(InsufficientFunds) method withdraw ($amount) { if ( ( $self-&gt;balance - $amount ) &lt; 0 ) { InsufficientFunds-&gt;throw(&quot;Naughty, naughty&quot;); } $self-&gt;next::method($amount); } } class Bank::Account::Premium :isa(Bank::Account) { # no need to override any behavior, but we could # provide more behavior as needed }  Seems like more work up front, but we have known for a long time that fixing broken code in the design stage is far cheaper than fixing it after it’s in production. But we’re so used to violating encapsulation, often without even realizing it, that we fall back on this rather than using proper design.  Yes, this example is contrived, but it’s based on my decades of OO programming. I also started off as a rubbish OO developer, having learned the syntax and not the theory. My punishment was spending years working with multiple clients, cleaning up the messes of other developers with the same background.  Conclusion  Encapsulation has been accepted best practice for OOP developers for decades. For Perl developers, it’s a contentious issue, due in part to how easy it is to apply “quick fixes” without having to revisit our design. And then, everyone quotes Larry. Larry Wall famously said  :   Perl doesn’t have an infatuation with enforced privacy. It would prefer that you stayed out of its living room because you weren’t invited, not because it has a shotgun.   That might work for disciplined developers who understand OOP. But we’re often undisciplined. We often don’t understand OOP. We’re often under time pressure and it’s easy to make mistakes then.  Larry has done great things with Perl, but this saying has outlived its usefulness. For small, quick hacks, or proofs-of-concept, that’s fine. But as systems grow, it’s not fine. There’s a saying that wise men make saying so fools can repeat them. I have long been one of those fools, but years of fixing broken OOP systems in many shops have taught me that I was wrong.  Corinna likely will allow encapsulation violation via a cumbersome meta-object protocol (MOP), but this will glaringly stand out like a sore thumb as a code smell. It will be clear that there’s a design flaw and the design should be revisited.  Not being able to easily violate encapsulation seems like a burden, but only because we’ve grown lazy and have skipped thinking about design. In reality, it will help us build more robust systems and teach us when our design is flawed. Far from being a limitation, encapsulation will be a powerful tool.  ","title":"Understanding Class Inheritance","url":"\/articles\/understanding-class-inheritance.html"},{"body":"    What’s an Object?  Generic Experts  Mutability  Object-Relational Mappers  Teaching OOP?  Conclusion     Disclaimer : I’m the lead designer of the Corinna object-oriented system going into the Perl core  . I’m probably a bit biased here.   It seems like a week can’t go by without another blog entry or video explaining why object-oriented programming (OOP) is bad  . While the content of many of those articles is bad, if you read enough of them, some common themes emerge: “mutability” and “genericity.” Both are difficult to solve, so it’s worth explaining what’s wrong here. But first, what’s an object?  What’s an Object?  Let’s take an extended quote from my book Beginning Perl  .   ÆVAR THE PERSONAL SHOPPER  You’re an awfully busy person and have little free time but plenty of disposable income, so you’ve decided to hire a personal shopper. His name is Ævar (any resemblance to reviewers of this book, living or dead, is purely coincidental) and he’s friendly, flamboyant, and most of all, cheap.  Because Ævar is new to both your city and the job, you have to tell him carefully how much money he can spend, exactly what quality of products you want, and where to buy them. You may even have to tell him which route to drive to pick up the goods and how to invoice you.  That, in essence, is procedural code and that’s what you’ve been doing up to now. You’ve been carefully telling the computer every step of the way what to do.  After a few months of explaining every little detail, Ævar gets upset and says, “þegiðu maður, ég veit alveg hvað ég er að gera” (Icelandic for “Shut up dude; I know what I’m doing”). And he does. He knows what you like and where to get it. He’s become an expert. In OO terms, you might now be doing this:  my $aevar = Shopper::Personal->new({ name => 'Ævar', budget => 100 }); $aevar->buy(@list_of_things_to_buy); my $invoice = $aevar->get_invoice;  You’re no longer telling Ævar every little step he needs to take to get your shopping done. He’s an expert, and he has all the knowledge needed to do your shopping for you and present you with the bill.   That’s all objects are. They are experts about a problem domain, but that’s actually a problem.  Generic Experts  This issue is extremely important, but it’s one that’s not touched on often enough.  OOP works best in constrained environments. For a company, if your developers really understand OOP (they often don’t), OOP is great because you can create custom experts for your domain. Your objects know how your custom authentication and authorization system work, so you can share this code with other developers in the company and they don’t have to rewrite it.  Until that one team has some custom authorization rules dealing with SOX compliance, GDPR, PCI, or a host of other things you’ve never heard of. They might ask you to make your objects more “generic” to allow them to handle their custom needs, but that starts to add complexity and, potentially bugs. If enough teams ask for this, your beautiful authorization object can become an unmaintainable god object. Systems grow. Needs grow. And software often suffers as a result.  This is fundamentally a problem with abstraction . Just because I develop a good abstraction for my use case doesn’t mean I’ve developed a good abstraction for yours.  In other words, for generic OOP, it can often be problematic because your solution may not be general enough to fit other people’s needs. For Perl, the LWP  objects to handle Web communication tend to work very well because it’s a well-constrained problem space.  In contrast, my own HTML::TokeParser::Simple  module is less generally useful because it fits a particular use case that doesn’t map well to many closely-related problem spaces, For example, when you need to map out the hierarchical structure of HTML, the stream of tokens my module produces aren’t well=suited to this. Thus, you may find yourself using a class that kinda works for what you want, but is ill-suited for what you need.  Another interesting problem is dealing with scale. For example, in a small company with maybe 20 to 30 programmers, a well-designed object can be a godsend. In a company with 200 to 300 programmers, a well-designed object can often be a source of pain because it doesn’t quite fit the needs of other teams. However, it’s often hard to make an object extensible in a way that fits the needs of others because it’s hard to plan for needs that you’re not aware of. If you’re lucky, your company might have a policy that all shared objects have a shared interface (possibly with default implementations) and different teams implement them to suit their unique needs.  If you’re unlucky, different teams all implement their own Customer class and if your code interacts with their code, there’s no guarantee of interoperability. I speak from bitterly personal experience where another team implemented a User object that had only the two methods they needed, not the many, many more that were generally available. A search through their multi-million line codebase revealed many custom-made user objects, none of which really worked like the others, but any of which could potentially leak into other’s code (this is where static typing or type constraints really shine, but that’s another article).  Objects for poorly-defined problem spaces don’t scale. If you might face this, define those problem spaces before the problem starts. Let people create their own custom objects, but with the minimum required common functionality.  Mutability  The big issue that is touched on, however, is a serious problem: mutable state. You can send an object to many different areas of your code and then some code far away from you mutates the state and suddenly you find that the DateTime object you were using has a different state from what you had tested earlier.  As a general rule, sending an object somewhere is sending a reference, not a copy. Why is this bad? You can read Ricardo Signes' horror story about a bug dealing with mutable state in objects  . When you’re building a complex system and you cannot depend on the state of your objects, you are in for a world of hurt when it bites you because it can be a nightmare figuring out what unexpectedly changed the state.  This is irrelevant in the face of immutable objects. Once you fully understand why changing an object’s state is bad, your programming becomes much more robust.  If you would like to know more about the issues with immutable objects, read my why do we want immutable objects? article.   As an aside, if you switch to a service-based architecture , you can’t send references, only copies. This makes the system more robust, but the trade=off is performance. This is why cloning is often shallow instead of deep. But , if you have immutable objects, sending a reference is irrelevant because you don’t risk mutating its state. Well, unless you’re using an ORM and something outside the program mutates that state. Speaking of which ...   Object-Relational Mappers  But then we come to the real problem: Object-Relational Mappers (ORMs). ORMs are almost by default mutable (there are immutable ORMs, but they’re not common).  my $customer = Acme::Model::Data::Customer-&gt;new($id); $customer-&gt;name($new_name); $customer-&gt;save;  This is real fun when separate parts of the code fetch the same object from the database and mutate it at the same time. Optimistic offline locking  and similar strategies are your friend here.  Otherwise, you’re passing those around, it’s trivial for something to mutate the state and cause you grief. For Tau Station  , we (I) learned this the hard way and now we often fetch data from the database and simply return a copy of the data structure. It’s faster and safer that way.  But regardless of the safety of protecting the underlying data, it still doesn’t alter the fact that not only is the state mutable, it’s stored outside the program. You can be as careful as you want, even creating read-only instances of ORM objects, but if two processes pick the same object and something changes the underlying state, you can still have incorrectly functioning software. ORMs make working with databases much easier, but there is a price to pay.  Teaching OOP?  Perhaps the biggest obstacle to effective OOP is, sadly, teaching it.  When I was learning Java in college, I still remember my first Java instructor, fresh out of college, struggling to explain the different between classes and instances. You could tell she knew the difference, but she had trouble articulating it in a way that students could understand. But that’s class-based OOP. Not all OOP systems are based around classes. Prototype-based OOP  doesn’t have classes.  My second Java instructor had us build a Swing interface to read some records from a database and display them in a form. I immediately thought MVC and built model, view, and controller classes, along with a complete set of JUnit tests. My instructor rejected my code on the grounds of “I think this code might be good, but I don’t understand it. Please submit all of it as a single class.” I also had to not submit my JUnit tests.  Admittedly, my two Java instructors constitute an anecdote and not information you can actually act on, but I’ve seen this again and again why people try to explain what objects are.   Objects are data with behavior attached  Objects are smart structs  Objects are user-defined data types  Objects are polymorphic containers  ... and so on   All of those miss the point. They’re talking about the implementation or structural details of objects and not the reason for objects. Why do we have cars? So we can drive from place to place. We can argue about the engines later.  The reason we have objects, as I noted above, is that object can be experts about a problem space. I can hand an object to a junior developer who knows nothing about the underlying complexity and that junior developer can use that underlying complexity . Pretty cool, huh?  Using objects is great, but building objects? Learning how to do this well involves a bewildering array of concepts such as SOLID  , GRASP  , the Liskov substitution principle  , the Law of Demeter  , not exposing implementation details (also known as the abstraction principle  ), and so on. The last point, I might add, is why I have objected to lvalue attribute in Corinna (unless we can also have lvalue methods). You should be able to freely convert this:  field $public_key :reader :writer;  To this:  method public_key ($optional_key = undef) { # fetch key from keystore }  And your interface remains unchanged on the outside of the class, allowing you to keep your contractual promises with the class' consumers.  But many developers have asked for this:  $object-&gt;public_key = $public_key; # field # versus this: $object-&gt;public_key($public_key); # method  For the above, we’ve tightly coupled the public interface to the private implementation just to add a little syntactic sugar.   Object-oriented design is confusing enough that my friend Bob Gregory  , coauthor of the book Architecture Patterns with Python: Enabling Test-Driven Development, Domain-Driven Design, and Event-Driven Microservices  , shared an anecdote of how he taught better design to developers. He got fed up with seeing classes with names like StartHtml (which would build the start of an HTML document), that he mandated classes be named after creatures . He wound up with classes like the AuthenticationFairy and the SessionGnome , experts in their particular domains. Weird, but it appeared to work.   Since many OOP developers are self-taught, and with many OOP implementations being hard to learn, and the bewildering array of complex and sometimes contradictory “best practices” we have to learn, OOP seems to be a more advanced concept than many developers realize.  Conclusion  I think there’s a good case to be made that OOP is not nearly as useful as it’s claimed. I have worked with far too many clients whose OOP systems are, quite frankly, broken. I’ve spent a considerable about of time and energy (and thereby client money) fixing these broken systems. For one, someone had created a system whereby every time you created a new system component, you had to create a new class holding information about that component. No specialized behavior—just information. In fact, for each component, sometimes you had to add more than one new class—again, just for storing information (almost all of it strings). I proposed a different, data-driven system which, when done, would replace about 30 (and growing) classes with two (not growing).  In contrast, great OOP systems are a joy to work on and have made my life easier. I strongly recommend them, but they don’t seem terribly common.  In the research study “Productivity Analysis of Object-Oriented Software Developed in a Commercial Environment”  (Potok, Vouk, Rindos, 1999), the researchers found that there is very little quantitative evidence that OOP improves productivity in commercial environments. On the plus side, there was also little evidence that it hinders productivity. And I know that when I hand a hideously complex authorization object to a new developer, so long as they stick to the interface, it magically works . I call that a win.  I love programming in OOP and (mostly) prefer it over procedural code (we’ll skip other paradigms for now). Further, a great many programmers seem to reach for it as a first solution rather than the last, so I’ll continue to work on the Corinna OOP system to make OOP easier for Perl, but I’m hard-pressed to combat criticisms against OOP itself. For now, just remember:   Immutability is your friend  Keep your interfaces small  Well-constrained problem spaces are better for OOP  Poorly-constrained problem spaces can be great for OOP, but they don’t scale   There’s a lot more than that, but those are great starters for decent OOP programming.  Until next time, happy hacking!  ","title":"Why is Object-Oriented Programming Bad?","url":"\/articles\/why-is-object-oriented-programming-bad.html"},{"body":"    What’s an Object in Perl?  Types and Objects  Why not Moose?  Why Corinna?     As you may know, we’re designing a new OO system for the Perl language . A couple of people keep asking why we don’t just put Moose  in the Perl core.  Please note that no one is taking away bless or Moose or anything like that. If you prefer these tools, that’s fine. After all, many people strongly objected to Moose when it first came out, echoing the arguments I hear against Corinna (except that Moose was\/is slow. Corinna is not, despite the fact that no optimization has been done on the Object::Pad prototype). It took years before Moose (and later Moo) won the hearts and minds of Perl OOP developers.  So the following explanation is not for the those dead-set against Corinna. It’s for everyone else who is curious about why Perl might adopt the Corinna proposal for native object-oriented programming (hereinafter referred to as OOP), even when the Perl 5 Porters (P5P) have rejected Moose.  What’s an Object in Perl?  If you know Perl, the answer to the above question is “nothing.” Perl knows nothing about OOP. It doesn’t know what objects are. Instead, there are a few features introduced in Perl 5  to give one a basic toolkit for emulating some OOP behaviors.   An object is a data structure that knows to which class it belongs.  A class is the same thing as a package.  A method is subroutine that expects a reference to an object (or a package name, for class methods) as the first argument.  You inherit from a class by adding its name to your namespace’s @ISA array.   Does that sound like a kludge? It is, but it’s worked well enough that developers without much experience in OOP have accepted this.  It’s actually kinda rubbish because it’s more or less an assembler language for OOP behavior, but it works. In fact, it’s modeled after Python’s original implementation of objects, but with changes to take into account that we are not, in fact, Python.  So imagine a Person class written in core Perl. You have a read\/write name and that’s all. We’ll keep it simple. This is what you used to see in OOP Perl.  package Person; use strict; use warnings; sub new { my ( $class, $name ) = @_; return bless { name =&gt; $name } =&gt; $class; # this is the instance } sub name { my $self = shift; if (@_) { $self-&gt;{name} = shift; } return $self-&gt;{name}; } 1;  So you want to subclass that and create a Person::Employee and need to keep their salary private:  package Person::Employee; use strict; use warnings; our @ISA = 'Person'; sub new { my ( $class, $name, $salary ) = @_; my $self = $class-&gt;SUPER::new($name); $self-&gt;{salary} = $salary; return $self; } sub _salary { # the _ says &quot;private, stay out&quot; my $self = shift; return $self-&gt;{salary}; } 1;  OK, kinda clumsy, but it works. However, if you want to see the salary:  use Data::Dumper; print Dumper($employee);  Admittedly, salary is often very secret and the above code is rubbish, but there’s plenty of other data that isn’t really secret, but you don’t want to expose it because it’s not part of the interface and people shouldn’t rely on it. But what’s going on with that leading underscore?  In Python, you won’t get a method for that, but if you know how the method names are mangled under the hood, you can jump through hoops and call them. For Perl, you just have that method. Developers know they shouldn’t call them, but I investigated three code bases from our various clients and found:   Codebase 1: very good Perl, lots of tests, tremendous discipline, and 72 calls to private methods outside the class (or subclass) they were defined in)  Codebase 2: very sloppy Perl, few tests, and an organizational mess, but only 59 calls to private methods (to be fair, it was a smaller codebase, but not by much)  Codebase 3: a combination of miserable and wonderful Perl, almost no tests, and the largest code base we have ever worked with. About 20,000 calls to private methods.   “Private” methods in Perl are not private. At all. You have a deadline, you have a very hit-or-miss codebase, and you put something together as fast as you can. All of a sudden, the “private” code isn’t very private any more. Work with a large group of programmers with varying levels of ability? They’re going to call those private methods, they’re going to call $customer-&gt;{name} instead of $customer-&gt;name , and do all sorts of other naughty things.  What this means is that your private methods are now part of the public interface whether you like that or not. You might think you can change them, but you do so at the risk of wide-spread carnage.  If there is one things I’ve learned in decades of programming for multiple clients, it’s this:   Relying on developers to “do the right thing” is a disaster if the wrong thing is easy to do.   Corinna is designed to scale .  Types and Objects  Before we get to the discussion of why Moose isn’t going into the core, let’s consider types (we’ll skip type theory).  Types have names, allowed values, and allowed operators.  For many programmers, when they think of types, they think about int , char , and so on. Those are the names of the types.  Types have a set of values they can contain. For example, unsigned integers can hold zero and positive values up to a certain limit (determined by the amount of memory assigned to that type). You cannot assign -2 to an unsigned integer.  You also have a set of allowed operations for those types. For example, for many programming languages, multiplying the string “foo” by a number is a fatal error, often at compile time. However, Perl allows this:  $ perl -E 'say &quot;Foo&quot; * 3' 0  Of course, you should almost always enable warnings, in which case you’ll see a warning like this:  Argument &quot;Foo&quot; isn't numeric in multiplication (*) at -e line 1.  Or if you prefer, you can make these kinds of errors fatal:  use warnings FATAL =&gt; &quot;numeric&quot;;  Types such as int , char , bool , and so on, are largely there for the computer. They usually map directly to things the CPU can understand.  But what does this have to do with objects?  Objects have names (the class), a set of values (often complex), and a set of allowed operations (methods).  These map to problem domains the programmer is concerned with. In other words, they’re complex types written to satisfy developer needs, not computer needs. They’re not just a grab-bag of miscellaneous features that have been cobbled together ... well, they currently are for Perl.  Now let’s look at Moose.  Why not Moose?  First, Moose isn’t going into core because P5P said “no.” It pulls in a ton of non-core modules that P5P has said they don’t want to maintain. That should end the argument, but it hasn’t.    Truly a majestic beast.   Some argue for Moo in core, but Moo has this in its documentation:   meta  my $meta = Foo::Bar-&gt;meta; my @methods = $meta-&gt;get_method_list;  Returns an object that will behave as if it is a Moose metaclass object for the class. If you call anything other than make_immutable on it, the object will be transparently upgraded to a genuine Moose::Meta::Class instance, loading Moose in the process if required.   So if we bundle Moo we have to bundle Moose, or break backwards-compatibility on a hugely popular module.  But there’s more ...  As stated earlier, objects in Perl are:   An object is a data structure that knows to which class it belongs.  A class is the same thing as a package.  A method is subroutine that expects a reference to an object (or a package name, for class methods) as the first argument.  You inherit from a class by adding its name to your namespace’s @ISA array.   Moose is simply a sugar layer on top of that. Native objects in Perl have no understanding of state or encapsulation. You have to figure out how to cobble that together yourself.  Here’s the person class (using only core Moose and no other helper modules):  package Person; use Moose; has name =&gt; ( is =&gt; 'rw', # it's read-write isa =&gt; 'Str', # it must be a string required =&gt; 1, # it must be passed to the constructor ); __PACKAGE__-&gt;meta-&gt;make_immutable;  Right off the bat, for this simplest of classes, it’s shorter, it’s entirely declarative, and it’s arguably more correct. For example, you can’t do $person-&gt;name(DateTime-&gt;now) as you could have with the core OOP example. You could fix that with the core example with a touch more code, but you’d have to do that with every method you wrote.  You have to change construction a bit, too. Either of the following works:  my $person = Person-&gt;new( name =&gt; 'Bob' ); my $person2 = Person-&gt;new({ name =&gt; 'Bob2' });  Why two different ways? Who knows? Live with it.  But what about the subclass?  package Person::Employee; use Moose; BEGIN { extends 'Person' } has _salary =&gt; ( is =&gt; 'ro', # read-only isa =&gt; 'Num', # must be a number init_arg =&gt; 'salary', # pass 'salary' to the constructor required =&gt; 1, # you must pass it to the constructor ); __PACKAGE__-&gt;meta-&gt;make_immutable;  And to create an instance:  my $employee = Person::Employee-&gt;new( name =&gt; 'Bob', salary =&gt; 50000, );  And we don’t have a -&gt;salary method, but we can still access -&gt;_salary . Hmm, not good. That’s because Moose manages state, but doesn’t make it easy to provide encapsulation. But at least it protects the set of allowed values.  try { $employee-&gt;name( DateTime-&gt;now ); say $employee-&gt;name; } catch ($error) { warn &quot;Naughty, naughty: $error&quot;; }  And that prints something like:  Naughty, naughty: Attribute (name) does not pass the type constraint because: Validation failed for 'Str' with value DateTime=HASH(0x7ffb518206d8) at accessor Person::name (defined at moose.pl line 11) line 10 Person::name('Person::Employee=HASH(0x7ffb318d47b8)', 'DateTime=HASH(0x7ffb518206d8)') called at moose.pl line 42  Oh, but it actually doesn’t fully protect that set of values:  try { $employee-&gt;{name} = DateTime-&gt;now; say $employee-&gt;name; } catch ($error) { warn &quot;Naughty, naughty: $error&quot;; }  Note that we’ve reached inside the object and set the value directly. The above code cheerfully prints the current date and time.  Naturally, you can set the salary the same way. You don’t want people messing with private data.  And finally, Corinna.  class Person { slot $name :reader :writer :param; }  Hmm, that looks pretty easy, but what about that ugly salary problem?  class Person::Employee :isa(Person) { slot $salary :param; }  Now, internally, everything has access to $salary , but nothing outside the class does. It’s no longer part of the public API and that’s a huge win. You literally cannot set it from outside the class.  It’s also great that $salary is just a local variable and we don’t have to keep doing method lookups to get it. With Paul Evan’s   Object::Pad  test bed for Corinna, that’s been a huge performance gain, despite little work being done on optimization.  But sadly, we don’t yet have full safety on the domain of values. There’s been some discussion, but to make that work, we need to consider types across all of the Perl language. That means variable declarations, slot declarations, and signatures. We’re not there yet, but already we have something better. This, admittedly, is the biggest downside of Corinna, but we have a more solid foundation for OOP.  Why Corinna?  To be honest, encapsulation isn’t very compelling to many Perl developers. In fact, many of the best things about OOP software isn’t compelling to Perl developers because Perl doesn’t seem to have many OOP programmers with OOP experience outside of Perl, so it’s hard for them to appreciate what is missing.  As a case in point, here was a complaint from someone on the Perl 5 Porter’s mailing list that echos complaints made elsewhere:   Rather than peeling the OOP “onion” back layer by layer, build it out from what exists now. Starting with what’s needed to augment “bless”, prototypes, and overload.pm.   The problem, at its core, is that this misunderstands the problem space and current attempts to fix this.  The following is an edit of the response written by Chris Prather which nicely sums up some of the problems.   Without trying to sound offensive, this list kinda suggests you’ve not really done any extensive thought about what an object system is and should be. Most people don’t and shouldn’t ever need to. A list of things that in my opinion would need enhancement:   Classes: Perl just gives you packages with a special @ISA variable for inheritance. Packages are just a bag of subroutines, they have no idea of state.  Attributes: bless associates a package with a data structure to provide “attributes”, except it doesn’t actually provide attributes, it just provides a place to store data and leaves you to figure out what attributes are and what that means. This also means that all instance data is public by default. While we pretend that it doesn’t because Larry told us not to play with shotguns, it hasn’t stopped a lot of people putting shotgun-like things onto CPAN (or into Perl Best Practices).  Metamodel: The way you interrogate and manipulate a package is ... not obvious. Package::Stash exists on CPAN simply to provide an API for this manipulation because it’s fraught with edge cases and weird syntax.  Methods: Perl’s concept of a method is a subroutine called in a funky way. Combined with the public nature of the data, this means you can call any method on any object ... and the only thing that can prevent this is the method itself. I’ve never seen anyone write enough validation code at the beginning of their methods to deal with what is actually possible to throw at a method.  Class composition: Design Patterns: Elements of Reusable Object-Oriented Software , published literally 4 days after Perl 5.000 says to prefer composition to inheritance. Perl’s only solution to reusable behavior is inheritance. Worse, Perl supports multiple inheritance using a default algorithm that can cause weird, non-obvious bugs.  Object Construction Protocol: Ensuring that all of the attributes are initialized properly in the correct data structure during construction is left entirely as a lemma for the programmer.  Object Destruction Protocol: See above, but because Perl has universal destruction where we can’t even guarantee the order in which things are destroyed.   The fact that Perl’s built in object system just gives you a bag of primitives and leaves you to build a robust object system for every application you write is kinda the reason things like Moose exist. Moose’s choices to solve many of these problems is the reason Corinna exists. Let’s take Classes, attributes, and methods for example (because this is the most obvious change in Corinna). Classes are supposed to be a template for creating objects with initial definitions of state and implementations of behavior. Perl’s native system only provides the second half of that.   Ultimately, by using Corinna, we can have Perl know what an object type is, not just the current hodge-podge of SVs, AVs, HVs, and so on.  Subroutines and methods will no longer be the same thing, so namespace::clean and friends become a thing of the past.  We can eventually write my $object = class { ... } and have anonymous classes.  Corinna (via the Object::Pad test bed) already gets the tremendous benefits of compile-time failures if $name is missing, instead of the run-time failure of $self-&gt;name not existing.  We can have compile-time failures of trying to call instance methods from class methods, something we’ve never had before. In fact, there are many possibilities opened up by Corinna that you will never have with Moo\/se.  That’s why Moose isn’t going into core. It also gives insight into why we’re not trying to gradually improve core OOP but are instead jumping right in: you can’t get there from here because they’re fundamentally different beasts. You can’t take a handful of discrete primitives and suddenly transform them into a holistic approach to OOP. Maybe someone could map out a way of doing that in an incremental fashion and by the year 2037, the last three Perl programmers will have a perfectly reasonable OOP system. I am tired of waiting.  I’m not saying Corinna is going to make it into core, but the prospects for it look very good right now. Corinna opens up a world of possibilities that Moo\/se can’t give us because Moo\/se is trapped within the constraints of Perl. Corinna is removing those constraints.  ","title":"Why is Perl not putting Moose in the core?","url":"\/articles\/why-is-perl-not-putting-moose-in-the-core.html"},{"body":"    Introduction  The Routine  Starting the Day  Get Dressed  Set a Schedule  Starting Work  Communicate  Be Human!  Taking Breaks  Ending the Day  Conclusion     Introduction  By now, with the COVID-19 pandemic sweeping the world, many companies are turning to remote work for the first time. We’ve been having conversations with some of them, and there’s been quite a spike in traffic for my article about managing remote teams . Read that article if you want to know how to build and run remote teams. Read this article if you want to know how to be productive when remote.     We say what we do and do what we say.    Note that the following, along with the managing remote teams teams article, is a product of years of refinement because All Around the World  has been doing this for years. Whether you’re a backend Perl hacker, a front-end UI genius, or deep in the trenches of DevOps (hint: we do all of these and more), the process is the same.  But frankly, most articles about working from home are boring and sound like they’re written by that creepy guy from HR whose permanent smile never quite reaches his eyes.  Instead, let’s look at how your day goes.  The Routine  Starting the Day    Make that bed! Source    According to the Center for Disease Control, 1 in 3 adults don’t get enough sleep.  . Not only does this impact your health, it impacts your productivity at work. So get a good night’s sleep. But how?  First, get rid of your damned snooze button!  I used to hit snooze multiple times every morning, sometimes being in a frantic rush to get to work on time because I’d have hit that snooze button five or six times. In fact, I would set my alarm clock to wake me up too early because I knew that snooze button was laying in wait for me.  Every alarm clock with a snooze button I’ve ever had has a way to turn that button off. Do it. It might sound scary at first, but when you wake up and realize that there is no snooze available, you have no choice but to get up.  I was surprised at how well this simple trick works and I’ve been using it for years.    Next, when you get up in the morning, make your bed. I doesn’t have to have perfectly crisp “hospital” corners, but it should be comfy enough that when you get back to bed, your first thought isn’t about how messy and uncomfortable the bed is. Instead, it’s just lifting the covers and sliding into bed for a good night’s sleep.  Not only are there studies showing numerous benefits associated with making your bed  (with the caveat that correlation doesn’t equal causation), but it also sets the tone for the day: I’m going to do something productive.  Oh, and if you have trouble falling asleep, read this  .  Get Dressed  Don’t work in your pajamas!  This sounds silly, but after you get out of bed, take the time to brush your teeth, shower, shave (if you do), get dressed, and so on. The power of ritual and tradition is well-known for establishing positive connections and relationships. If you start work in your pajamas, you’re breaking a personal ritual (even if it doesn’t feel like one). So get up and act like you’re preparing for your commute, even if that commute is only to your home office.  The jokes about people who work from home developing poor hygiene are based in reality. On various forums, I’ve seen plenty of people lament that they’ve gotten careless. When I first started working remotely in the early 2000s, I did the same thing and discovered it was not a recipe for success.  Set a Schedule    Photo by Jon Tyson on Unsplash Source    Did you work 9 to 5 in the office? Great! Do it from home. Setting and maintaining a regular schedules is the single most important tip I can give you. Without the discipline of a regular schedule, it’s easy to get “sloppy” in the hours we work. If you’re allowed to work from home, don’t abuse the privilege. However, keep in mind that you might be able to adjust your schedule, assuming your company is understanding (some will insist upon 9-to-5 without appreciating the benefits of flextime).  Do you like to sleep in? Go ahead! If you want to work from 11-to-7, so be it. Just make sure that it’s regular and you’re still reachable by your colleagues. If you find that the best communication for your company happens after lunch, why not optimize for that and throw away the alarm clock?  Or are you a seriously early riser? Working from 6 to 2 has many advantages. Having a few early hours where you’re not constantly getting bombarded by communication from your colleagues can be very productive. And stopping work at two in the afternoon feels positively sinful . Even if you can only do that one day a week, see if you can squeeze it in. It’s great for Fridays!  Whatever hours you keep, even if you vary them, it’s best if you have a schedule posted that you can share with your colleagues. If they’re frantically pinging you on Slack and don’t realize you’re sitting on the terrace reading a book because you’re off work, things will go bad.  Starting Work  When you’re sitting down with your mug of coffee (never tea, you heathens!) at your computer, in your “work area” that doesn’t overlap your personal area (if possible), be sure to hop on Slack, IRC, or whatever your company uses for quick chatting and greet everyone with a cheery “good morning!” This sets a positive tone and let’s everyone know you’re there.  But there’s a far more important reason for doing this. When we work remotely, it’s easy to get isolated from other people, to lose those connections. Instead of just tucking straight into work, some invisible cog in some invisible machine, you’re reaching out and talking to people and reforging social bonds that are so easy to break when you’re at home all the time.  And then you start work.  But wait! There’s a trap. Quite often that “starting work” means checking Facebook, reading the news, and so on. You do that before you start working. Set your alarm earlier if you need to.  Instead, if you find yourself having trouble starting working, find a small task and do it. An active task (not “read company email”), not a passive one. Quite often you’ll find that just actively doing one small “positive” thing for your work will jump start your motor and get you going. That, and coffee.  Communicate  This is overwhelmingly your biggest challenge. Plenty of people find they can be productive while working remote, but when you’re trying to figure out how to untangle a huge mess of code you’ve been tasked with fixing, you can’t just lean over the cubicle wall and start an in-depth conversation with Sarah about best approaches. Unfortunately, this can teach you to be too self-reliant.  Small tasks? No big deal? Big tasks? Big deal. You want this knowledge-sharing. You want to discuss different approaches. You want other people to realize that changes are coming down the pike. But getting into a personal bubble and not communicating is one of the known problems with remote work.  So make sure you communicate well and often. And choose the most effective communication tools you can find. Dr. Alistair Cockburn, in his presentation Characterizing people as non-linear 1st order components in software development    , found that communication effectiveness, from best to worst, tends to be:     Face-to-face  Video  Audio  Email  Paper   Given that “face-to-face” is off the table, video is clearly the next best option. I can’t tell you how many times in a group video call I’ve seen one person explain an idea and—sometimes it’s just a subtle shift of the eyes—another person gives off a non-verbal cue that they have reservations. It’s a non-verbal cue that you won’t have with “voice only” communications and once you get used to watching for them, you won’t want to go back to voice only.  Just to restate that: use video, use video, use video !  Got it?  For quick, transient communications, Slack, IRC, or whatever tool you use is fine. If you need something more permanent, email is good, though it’s often ignored. Learning to write effective email is a skill that many have skipped. But when you need to talk, use video.  Be Human!  While you’re on those video calls (especially for your daily-stand ups), don’t just talk about your TPS reports. You’re human. Make connections. Tell jokes. Swap war stories. Talk about your lives.  You don’t want to spend the entire meeting doing this, but you need a virtual water cooler and “slack chats” ain’t it. You want to see each other, if possible. And build those human connections. Your colleagues aren’t cogs churning out work; they’re human. If you want them to see you as you instead of as another cog, you need to give them the same respect.  Note: this is less important if you only work from home occassionally. Having the face-to-face time with colleagues makes all the difference in the world.  Taking Breaks    Take your breaks away from the computer Source    This one should be self-explanatory, but it’s not. Take your breaks. And don’t take them in front of your work computer.  Go for a walk. Move about. Do something physical instead of sitting in a chair and rapidly moving your fingers for eight hours.  And while we’re at it: don’t take lunch at your desk. You’re a human being and you need variety and physical activity. Physical activity can be as simple as walking, and there are multiple benefits to even small amounts of daily walking  .  Ending the Day  In the office, would you simply get up and walk out without saying goodbye to your colleagues? Just because we’re online doesn’t mean we ignore social norms. Let your colleagues know you’re off for the day! It’s a social activity, sets clear boundaries, and if someone is trying to reach you after, it’s more likely that one of your collegues will have noticed and let them know that you’re not there.  Conclusion  Working remotely isn’t the same as working in an office. It’s not inherently a social activity, and a five-second commute isn’t condusive to maintaining physical activity. Worse, it’s easy to slack off.  Being social, physical, and productive isn’t hard to do when you’re working from home, but you’ll need to be more mindful of it. Following the simple tips in this article will make that easier.  ","title":"Work From Home Effectively","url":"\/articles\/work-from-home-effectively.html"},{"body":"    Preface  What’s an Object?  Recommendations  Immutable Objects  Recommendation  Small Interfaces and Encapsulation  Recommendation  Type Library  Recommendation  Subclassing  Recommendation  Performance  Recommendation  Problems  Too Many Responsibilities  Recommendation  Use Methods for Attribute Data  Recommendation  Confusing Subroutines and Methods  Recommendation  Inconsistent Return Types  Recommendation  Ignored Constructor Arguments  Recommendation  Leftovers  Dr. Frankenstein  Hire Us     Preface  After spending almost three years as the lead designer of the Corinna object system for the Perl core  , I’m delighted that work on including this in the Perl language will start soon. Until it’s there, our company  still does a lot of work with clients, fixing their object-oriented code (OOP) and we’ve seen some very common causes of failures. Some say that our recommendations are too strict, and for personal projects, they might be. But we work with large-scale codebases, often written in Perl, and often containing over a million lines of code. When you’re working at that scale, especially when coordinating with many developers who may not know your code well, you need to be more rigorous.  This is not a tutorial on OOP design. That in itself is a complex topic (and perhaps one of the strongest arguments against OOP). So we’ll assume you need to fix an existing OOP codebase rather than design a new one.  Note: while these examples are in Perl, many of these problems apply to codebases we’ve worked on in other OO languages.  Warning : whenever we link to a module on the CPAN, please read the documentation of that module if you decide to use it. There are often caveats, such as with the “strict constructor” modules we mention later.  What’s an Object?  First and foremost, we must keep in mind what an object is. This is covered more in other articles , but for this one, just keep in mind that an object is an expert about a particular topic. Forget all of those other explanations about objects being “structs with behavior”, or “a reference to a data type that knows what class it belongs to.” Those are implementation details that don’t tell you objects are for . In my opinion, they even hinder people’s understanding of objects because they keep you focused on the wrong thing. Objects are experts on a particular problem space, that’s all.  Recommendations  We could talk about SOLID  , GRASP  , the Liskov substitution principle  , the Law of Demeter  , and so on, but while those are important, you can read about those on your own. Instead, we’ll cover recommendations and common problems. You’ll need to use your best judgment to decide how (or if) they apply to your codebase.   Note: if this article makes your eyes glaze over, but you love the Moose OOP system, you might skip ahead to Dr. Frankenstein to see a small module built on top of Moose that implements some of the recommendations in this article.   Immutable Objects  I’ve written before why immutable objects are good , so I won’t go into too much detail. Sometimes it’s not practical to have immutable objects (e.g., caches), but immutability should be your default position. In most languages, objects are passed by reference, not value. The more widely used an instance of your object is, if it’s mutable, the more likely that code “A” will change the state in a way that code “B” didn’t expect.  Recommendation  Even if you’re not convinced, try writing and using immutable objects. Like anything new, it can take some getting used to, but there’s a payoff there. Just don’t insist that everything be immutable. Your colleagues will hate you.  Note that if your object must return references to data, those are often mutable. However, you can declare those as immutable, too.  package Some::Class { use Const::Fast 'const'; # set up data here sub get_data_structure { my $self = shift; const my %hash =&gt; ( name =&gt; $self-&gt;name, giggity =&gt; $self-&gt;_something_else, ); return \\%hash; } }  In the above, the get_data_structure method returns an immutable data structure. Attempts to access unknown hash keys or change any of the values is a fatal error (use exists to test hash keys, of course). Some people call it overkill. Others call it safety. Your mileage may vary.  If you prefer, you can deeply clone the data structure before returning it. By presenting a copy, different consumers calling that method will always have the same results, but if the returned reference itself is shared, you could have issues. That being said, this is an issue with all code, not just OOP.  Small Interfaces and Encapsulation  The interface that your class presents is your “contract” with everyone who uses your class. You should design that interface with care, but you shouldn’t expose what you don’t need to. Using one of my “go to” examples of the Corinna OOP system that will be available in upcoming releases (but not in the immediate future), here’s a basic LRU (least recently used) cache:  class Cache::LRU { use Hash::Ordered; field $cache :handles(exists) { Hash::Ordered-&gt;new }; field $max_size :param :reader { 20 }; method set ( $key, $value ) { if ( $cache-&gt;exists($key) ) { $cache-&gt;delete($key); } elsif ( $cache-&gt;keys &gt;= $max_size ) { $cache-&gt;shift; } $cache-&gt;set( $key, $value ); # add to front } method get ($key) { if ( $cache-&gt;exists($key) ) { my $value = $cache-&gt;get($key); $self-&gt;set( $key, $value ); # add to front return $value; } return; } }  With the popular Moose OOP system for Perl, that would look something like this (skipping the sub bodies):  package Cache::LRU { use Moose; use Hash::Ordered; use namespace::autoclean; has '_cache' =&gt; ( is =&gt; 'ro', init_arg =&gt; undef, default =&gt; sub { Hash::Ordered-&gt;new }, ); has 'max_size' =&gt; ( is =&gt; 'ro', default =&gt; 20 ); sub set { ... } sub get { ... } __PACKAGE__-&gt;meta-&gt;make_immutable; }  For raw Perl, it might even look like this:  package Cache::LRU; use strict; use warnings; use Hash::Ordered; sub new { my ( $class, $max_size ) = @_; $max_size \/\/= 20; return bless { max_size =&gt; $max_size, _cache =&gt; Hash::Ordered-&gt;new, }, $class; } # more code here  For both Moose and core Perl, it’s hard to encapsulate your objects and minimize your interface. For both of those, you can call $object-&gt;{_cache} to get the underlying cache. For Moose, you can also call $object-&gt;_cache (it’s very cumbersome in Moose or Moo to not expose methods for what should be private data). This means that you have exposed that data, no matter how nicely you’ve asked people to “stay out of the kitchen.” This means that if someone is accessing your “private” data, if you want to switch your internal cache to use Redis, SQLite, or something else, you can easily break the code of others who are relying on it.  We almost always see $object-&gt;_private_method or $object-&gt;{private_data} in client codebases and that’s one of the first things we try to fix, if we have time. As a class author, you need to know you can safely change the internals of your object.  Recommendation  Keep your classes as small as you can and stop making accessors public by default. For now, this means prefixing methods with an underscore to make them “private by convention.” With Corinna, you simply don’t provide :reader or :writer attributes:  class Cache::LRU { use Hash::Ordered; field $cache { Hash::Ordered-&gt;new }; field $max_size :param :reader { 20 }; ...  In the above, you can read (but not write) the max size, but there’s no direct access possible to the $cache (this will be possible via the MOP, the meta-object protocol, but we do not want violating encapsulation to be a natural default. Hitting the MOP to violate encapsulation will stand out like a coal pile in a ballroom in code reviews).   Note: some Perl developers use “inside-out” objects to enforce encapsulation. The CPAN has Object::InsideOut  and Class::InsideOut  , amongst others. While it’s easy to use instances of these classes, they were cumbersome to write and sometimes buggy. As a result, we do not experience them often in client code.   Type Library  As systems grow, it’s very easy to have this problem:  sub get_part_by_id ($self, $id) { return $self-&gt;_inventory_schema-&gt;resultset('Part')-&gt;find($id); }  Most of the time that works, but sometimes you get no result. What happened? Maybe your primary key is a UUID but you’ve supplied an integer? Oops. So now you have to rewrite that:  sub get_part_by_id ($self, $id) { unless ( $id =~ qr\/^[0-9a-f]{8}(?:-[0-9a-f]{4}){2}-[0-9a-f]{12}$\/is ) { croak(&quot;ID ($id) does not look like a UUID.&quot;); } return $self-&gt;_inventory_schema-&gt;resultset('Part')-&gt;find($id); }  Do you really want to write that? Do you really want to debug that? (there’s a small bug in that example, if you care to look for it). If you use UUIDs frequently, you don’t want to write that again.  If you’ve used Moose, you know how easy it is to use type constraints:  has 'order_items' =&gt; ( is =&gt; 'ro', isa =&gt; 'ArrayRef[HashRef]', writer =&gt; '_set_order_items', );  Of course, if you spell the isa as ArrayRef[HasRef] , you won’t find out until runtime that you’ve misspelled it. For these and other situations, just create a type library as a centralized place to put your types and share them across your codebase as needed. If there’s too much code, focus on critical paths first.  Recommendation  Creating a type library for the first time can be daunting. Here’s a simple one, based on the excellent Type::Tiny  by Toby Inkster. It will cover most of your basic needs and you can extend it later with custom types, if needed.  package My::Personal::Types; use strict; use warnings; use Type::Library -base; use Type::Utils -all; use Type::Params; # this gets us compile and compile_named our @EXPORT_OK; BEGIN { # this gets us most of our types extends qw( Types::Standard Types::Common::Numeric Types::Common::String Types::UUID ); push @EXPORT_OK =&gt; ( 'compile', 'compile_named', ); } 1;  Using it is simple.  use My::Personal::Types qw(compile UUID); sub get_part_by_id ($self, $id) { state $check = compile(UUID); ($id) = $check-&gt;($id); return $self-&gt;_inventory_schema-&gt;resultset('Part')-&gt;find($id); }  In Moose, use these constraints to turn misspelled types into compile-time failures (and to get a much richer set of allowed types):  use My::Personal::Types qw(ArrayRef HashRef); has 'order_items' =&gt; ( is =&gt; 'ro', isa =&gt; ArrayRef [HashRef], writer =&gt; '_set_order_items', );  Despite the Type::Tiny name, the manual is quite extensive  . Go there for more information.  Subclassing  A subclass of a class is intended to be a more specialized version of that class. A Car isa Vehicle , or a Human isa Mammal . However, it’s easy to get this wrong under the pressure of deadlines, complex code bases, or just plain not paying attention. Is Item isa Product correct? Or is Product isa Item more correct?  Ultimately, the problem manifests itself in what I call the “person\/invoice problem”: Person isa Invoice . That makes absolutely no sense, but your software doesn’t know that . Your software only does what you tell it to do. It can’t evaluate semantics (at least, not yet), and if you write code that runs, but doesn’t make sense, that’s too bad.  In fact, inheritance is so problematic that some OOP languages disallow it altogether, favoring composition instead. Some only allow single inheritance, but provide alternatives (mixins, interfaces, traits, etc.). As a general rule, we recommend you use roles instead of parent classes:   Role::Tiny  (for any OO code)  Moose::Role  (for Moose)  Moo::Role  (for Moo)   There’s also my own Role::Basic  which can be used where Role::Tiny is appropriate, but the philosophy of that module is different  and presents somewhat different features.  Sometimes inheritance is the right thing to do. For example, in the Test::Class::Moose  xUnit framework, we have “test control methods” which run code before the class is instantiated, before each method is run, after each method is run, and after the test class has finished running. A test_setup method might look like this:  sub test_setup { my $test = shift; $test-&gt;next::method; $test-&gt;load_fixtures(qw\/Customer Orders\/); }  In the above example, the $test-&gt;next::method is used to call the parent test_setup to ensure that all setup is ready before you try to handle this class’s setup. In fact, you might have your test_setup call a parent test_setup which in turns calls its parent test_setup . This is common in xUnit testing and the order in which events fire is important. With roles, this is often done with method modifiers, but the order in which they fire is often dependent on their load order and that is not guaranteed. If you find yourself frequently using method modifiers in roles, you might want to think about inheritance to ensure that you have complete control over the sequencing of commands.  Recommendation  We should prefer composition or roles over inheritance. Composition is good when you clearly have an object to delegate to. Roles are good when you have some behavior which might apply to unrelated classes (such as serialization to JSON or XML).  There’s much more we could say about roles, but a full tutorial is beyond the scope of this article.  Performance  Don’t stress about performance, really. It’s not the code. It’s the database. It’s always the database. Unless it’s the network. Or it’s disk I\/O. Or, or, or ...  Steve McConnell, in his fantastic book Code Complete, 2nd edition, writes:   It’s almost impossible to identify performance bottlenecks before a program is working completely. Programmers are very bad at guessing which four percent of the code accounts for 50 percent of the execution time, and so programmers who optimize as they go will, on average, spend 96 percent of their time optimizing code that doesn’t need to be optimized. That leaves little time to optimize the four percent that really counts.   Unless you know, at design time, you’ll have performance critical code (don’t write ray-tracing software in Perl, folks!), design a great system and worry about performance only when it’s proven to be an actual problem. Devel::NYTProf  is your friend here. Be aware that benchmarking can be an arcane art.  But when we talk about performance, whose performance are we talking about? The software or the software developer? Here’s a little benchmark for you:  #!\/usr\/bin\/env perl use strict; use warnings; use Benchmark 'cmpthese'; sub use_a_loop { my @numbers; foreach my $i ( 0 .. 9 ) { $numbers[$i] = $i \/ ( $i + 1 ); } return \\@numbers; } sub direct_assignment { my @numbers; $numbers[0] = 0 \/ 1; $numbers[1] = 1 \/ 2; $numbers[2] = 2 \/ 3; $numbers[3] = 3 \/ 4; $numbers[4] = 4 \/ 5; $numbers[5] = 5 \/ 6; $numbers[6] = 6 \/ 7; $numbers[7] = 7 \/ 8; $numbers[8] = 8 \/ 9; $numbers[9] = 9 \/ 10; return \\@numbers; } cmpthese( 1_000_000, { 'use_a_loop' =&gt; \\&amp;use_a_loop, 'direct_assignment' =&gt; \\&amp;direct_assignment, } );  Do you think the loop or the direct assignment is faster? Do you really care? Well, it should be pretty clear that the loop is much easier to maintain. The direct assignment, however ...  Rate use_a_loop direct_assignment use_a_loop 970874\/s -- -50% direct_assignment 1923077\/s 98% --  Whoa! Directly assigning the data is twice as fast as the loop! If something like that is at the bottom of a tight loop and benchmarking shows that it’s a performance issue, then yes, switching from a loop to direct assignment might be an idea, but that would kill developer performance when it comes to maintaining that code. If you must do that, document it carefully, perhaps including a snippet of the code that this replaces.  Recommendation  Design the system well and don’t worry about performance while building it. Doing so runs the risk of optimizing code that doesn’t need to be optimized and possibly makes the code harder to maintain, raising long-term costs.  Instead, if your code is suffering performance issues, benchmark your code and find out where the real problems are. Here’s a video of Tim Bunce explaining how to profile your code:     Problems  The above were general recommendations for OO code, but now let’s talk about common problems we encounter with our clients.  Too Many Responsibilities  For our Project 500 contract, our client was providing online credit card services for a major, well-publicized event. However, they had a serious problem: their code could only handle 39 credit card transactions per second per server. For this event, they had a contractual obligation to improve performance by an order of magnitude . That’s right; they were required to have their code run at least ten times faster. We had two weeks to develop a proof of concept (spoiler: we succeeded).  In digging in, it turns out that they had developed an in-house OO system and an in-house ORM. We quickly discovered that a single “charge $x to the credit card” transaction generated almost 200 calls to the database! This was one of their major bottlenecks.  For our client’s ORM, every time a request was made it would gather a bunch of metadata, check permissions, make decisions based on whether or not it was using Oracle or PostgreSQL, check to see if the data was cached, and then check to see if the data was in the database. Instantiating every object was very slow, even if there was no data available. And the code was creating—and throwing away without using—hundreds of these objects per request.  We considered using a “pre-flight” check to see if the data was in the database before creating the objects, but there was so much business logic embedded in the ORM layer that this was not a practical solution given our time constrain. And we couldn’t simply fetch the data directly because, again, the ORM had too many non-ORM responsibilities built into it.  On another system, we had an immutable object (yay!) that had disk I\/O and heavy data validation every time it was instantiated. Yet that data never changed between releases, so I was tasked with caching the object data. I restructured to class to separate the validation of instance data and the setting of instance data. Then I added a few lines of code to the class to handle this and it worked like a charm, but my tests missed an edge case where some data wasn’t cached properly because one bit of data was set during validation. I had made the classic mistake of putting too much logic in that class.  To address this, I built a simple class to properly cache objects on web server restart and it cached the object for me. Not only did it work right this time, I now had a flexible in-memory cache for other objects. Further, because the cache internals were encapsulated, if we want to switch the cache out for Redis or something else, it becomes trivial.   .  Recommendation  Don’t have your objects try to do too much. The “single-responsibility” principle generally means there should only be a single reason to change a class. In the real-world, this is easy to miss, but it will save you much grief if you get this right.  Use Methods for Attribute Data  In old-school OOP code for Perl, you might see something like this:  package Customer; use strict; use warnings; use Carp 'croak'; sub new { my ( $class, $name, $title ) = @_; croak(&quot;Name required&quot;) unless defined $name; return bless { name =&gt; $name, title =&gt; $title, }, $class; } sub name { $_[0]-&gt;{name} } sub title { $_[0]-&gt;{title} } # more code here 1;  So far, so good. But the business rules state that if the customer has a title, they must always be referred to by both their title and name, never just one or the other. Developers keep forgetting (or don’t know) this business rule, but remember: your object is supposed to be the expert here and it won’t get it wrong, so the object should handle this responsibility.  Now you’re rewritten the class to remover the title accessor and to provide a name method to encapsulate this logic:  package Customer; use strict; use warnings; use Carp 'croak'; sub new { my ( $class, $name, $title ) = @_; croak(&quot;Name required&quot;) unless defined $name; return bless { name =&gt; $name, title =&gt; $title, }, $class; } # always prefix their name with a title if it exists sub name { my $self = shift; return $self-&gt;{title} ? &quot;$self-&gt;{title} $self-&gt;{name}&quot; : $self-&gt;{name}; } # more code here 1;  Again, this code looks correct, but by eliminating the accessor for our title attribute, if we were forced to subclass this, how do we override it, especially if it’s used elsewhere in the class? We could just overwrite the name and title values in the blessed hash of the subclass and that might be good enough, but if we need to convert an attribute to a method—as we did with name we can’t easily do that now.  Recommendation  This is one of the advantages of Moose and Moo. You automatically get method accessors for data attributes, so users of those OO systems will rarely notice this unless they also get in the bad habit of doing $self-&gt;{title} .  Otherwise, if you’re writing core Perl, just include simple accessors\/mutators for your data (prefacing them with a leading underscore if they should be private):  sub title ($self) { return $self-&gt;{title}; } sub set_title ($self, $title) { $self-&gt;{title} = $title; }  Whether you prefer to overload a method to be both an accessor and a mutator, or to return the invocant on mutation are arguments that are far beyond the scope of this article, so we would recommend following the current style of your codebase. If it doesn’t exist, define it and stick to it (predictable interfaces are fantastic!).  Confusing Subroutines and Methods  Your Moose\/Moo classes should generally resemble the following:  package Some::Class { use Moose; use namespace::autoclean; # class definition here # make_immutable is a significant performance boost __PACKAGE__-&gt;meta-&gt;make_immutable; }  I was writing some client software that could cache some objects (it’s a coincidence that I’ve had to do this so often) and for one edge case, it was good if we could clone the objects first. Many things cannot be easily cloned, so if the object provides its own clone method, we used that instead of simply returning the object from the cache. It looked sort of like this:  sub fetch ( $class, %arg_for ) { my $object = $class-&gt;_fetch_for_identifier( $arg_for{identifier} ); if ($object) { return $object-&gt;can('clone') ? $object-&gt;clone : $object; } else { return $class-&gt;_instantiate_and_cache($arg_for); } }  That failed miserably because some objects were importing a clone subroutine and because Perl doesn’t have a distinction because subroutines and methods (though it will when Corinna is implemented), clone was in the namespace and the can('clone') method returned true. We tried to call a subroutine as a method, even though it wasn’t.  Recommendation  Using namespace::autoclean  or namespace::clean  will remove the imported subroutines from the namespace and make it much harder for code to call subroutines as methods. Read the documentation for each to decide which will work better for you.  Note that as of this writing, there is an outstanding RFC for Perl to allow a lexical importing mechanism which should make this less of a problem in the future if it’s adopted.  Inconsistent Return Types  This isn’t just an OOP problem, but it’s a common issue we see in code in dynamic languages, so it’s worth mentioning here. If you’re going to return an array reference or a hash reference, you usually want to ensure that’s all you return. For example:  my $results = $object-&gt;get_results; if ($results) { foreach my $result ($result-&gt;@*) { # do something } }  But what if you forget the if ?  my $results = $object-&gt;get_results; foreach my $result ($result-&gt;@*) { # do something }  If get_results returns undef when there are no results, the code blows up. However, if it returns an empty array reference, the loop is simply skipped. You don’t have to remember to check the return value type (unless returning nothing is an actual error). This is easy to fix in methods, but be wary of a similar trap with attributes:  has 'results' =&gt; ( is =&gt; 'ro', isa =&gt; Maybe[ ArrayRef ], );  Recommendation  There are several ways to handle this, including defining coerced types, but one of the simplest and safest:  has 'results' =&gt; ( is =&gt; 'ro', isa =&gt; ArrayRef, default =&gt; sub { [] }, );  In the above, we drop the Maybe and default to an array reference if no argument is supplied to the constructor. With that, if they pass an arrayref of hashrefs for results , it works. If they don’t pass results at all, it still works.  However, if they pass anything else for results , including undef , it fails. That’s great safety because while you want the types you return to be predictable, it’s also good to allow the types you pass in to be predictable. There are definitely exceptions to this, but they should be used with care.  Ignored Constructor Arguments  Going back to our results example:  package Some::Class; use Moose; has 'results' =&gt; ( is =&gt; 'ro', isa =&gt; ArrayRef, default =&gt; sub { [] }, );  What happens if we do my $object = Some::Class-&gt;new( result =&gt; $result ); .  No matter what the type of $result is, the value is thrown away because by default, Moose and Moo simply discard unknown arguments to the constructor. If you misspell a constructor argument, this can be a frustrating source of errors.  Recommendation  Fortunately, the fix for this is simple, too: MooseX::StrictConstructor  . (Or MooX::StrictConstructor  for Moo):  package Some::Class; use Moose; use MooseX::StrictConstructor; has 'results' =&gt; ( is =&gt; 'ro', isa =&gt; ArrayRef, default =&gt; sub { [] }, );  Now, passing unknown arguments is a fatal error.  If you’re using traditional bless syntax, you’ll have to manually validate the arguments to the constructor yourself, but the type library we outlined above can be used.  Leftovers  There’s a huge amount more we can say, including useful design patters, being cautious with method modifiers in roles, when to use abstract classes instead of roles, but we’re starting to get into the long tail of code smells in object-oriented code bases, so perhaps those are good for another article.  Dr. Frankenstein  If you’ve gotten this far, congrats! One thing very helpful with OO code is to develop a set of guidelines on how you’ll build OO code and stick to it. We’re going to be Dr. Frankenstein and build our own object system out of the parts we have laying around. By ensuring everyone uses this object system, we can have greater consistency in our code.  Let’s pretend you’ve settled on the Moose OOP system as the basis of your own. You’d like to ensure several things are true and you’ve come up with the following list:   Unknown arguments to the constructor are fatal  It should be easy to see which attributes are or are not required in the constructor  Attributes should default to read-only  namespace::autoclean must always be used  You want signatures and types  The Carp module’s carp , croak , and confess functions should always be present  You want the C3 MRO (though you should avoid multiple inheritance)  Your work uses v5.22, so you’ll enforce those features   For this, you’ve decided that param should replace has if the parameter is required in the constructor, and field should replace has if the parameter is not allowed in the constructor. Let’s use Moose::Exporter  to set this up.  We won’t explain how all of this works, so you have some homework to do.   Again, we’re not saying the above is what you should do; we’re just giving an example of what you can do.   package My::Personal::Moose; use v5.22.0; use Moose (); use MooseX::StrictConstructor (); use Moose::Exporter; use mro (); use feature (); use namespace::autoclean (); use Import::Into; use Carp qw\/carp croak confess\/; Moose::Exporter-&gt;setup_import_methods( with_meta =&gt; [ 'field', 'param' ], as_is =&gt; [ \\&amp;carp, \\&amp;croak, \\&amp;confess ], also =&gt; ['Moose'], ); sub init_meta { my ( $class, @args ) = @_; my %params = @args; my $for_class = $params{for_class}; Moose-&gt;init_meta(@args); MooseX::StrictConstructor-&gt;import( { into =&gt; $for_class } ); warnings-&gt;unimport('experimental::signatures'); feature-&gt;import(qw\/signatures :5.22\/); namespace::autoclean-&gt;import::into($for_class); # If we never use multiple inheritance, this should not be needed. mro::set_mro( scalar caller(), 'c3' ); } sub field { my ( $meta, $name, %opts ) = @_; # default to read-only $opts{is} \/\/= 'ro'; # &quot;has [@attributes]&quot; versus &quot;has $attribute&quot; foreach my $attr ( 'ARRAY' eq ref $name ? @$name : $name ) { my %options = %opts; # copy each time to avoid overwriting # forbid setting `field` in the constructor $options{init_arg} = undef; $meta-&gt;add_attribute( $attr, %options ); } } sub param { my ( $meta, $name, %opts ) = @_; # default to read-only $opts{is} \/\/= 'ro'; # it's required unless they tell us otherwise $opts{required} \/\/= 1; # &quot;has [@attributes]&quot; versus &quot;has $attribute&quot; foreach my $attr ( 'ARRAY' eq ref $name ? @$name : $name ) { my %options = %opts; # copy each time to avoid overwriting if ( exists $options{init_arg} &amp;&amp; !defined $options{init_arg} ) { croak(&quot;You may not set init_arg to undef for 'param'&quot;); } $meta-&gt;add_attribute( $attr, %options ); } } 1;  With that, and the My::Personal::Types above, here’s a (silly) example of how to use this:  #!\/usr\/bin\/env perl use lib 'lib'; use Test::Most; package My::Names { use My::Personal::Moose; use My::Personal::Types qw( compile Num NonEmptyStr Str PositiveInt ArrayRef ); use List::Util 'sum'; # removed my namespace::autoclean param _name =&gt; ( isa =&gt; NonEmptyStr, init_arg =&gt; 'name' ); param title =&gt; ( isa =&gt; Str, required =&gt; 0 ); field created =&gt; ( isa =&gt; PositiveInt, default =&gt; sub { time } ); sub name ($self) { my $title = $self-&gt;title; my $name = $self-&gt;_name; return $title ? &quot;$title $name&quot; : $name; } sub add ( $self, $args ) { state $check = compile( ArrayRef [Num] ); ($args) = $check-&gt;($args); carp(&quot;no numbers supplied to add()&quot;) unless $args-&gt;@*; return sum( $args-&gt;@* ); } __PACKAGE__-&gt;meta-&gt;make_immutable; } my $person = My::Names-&gt;new( name =&gt; 'Ovid', ); is $person-&gt;name, 'Ovid', 'name should be correct'; ok !defined $person-&gt;title, '... and no title'; cmp_ok $person-&gt;created, '&gt;', 0, '... and a sane default for created'; ok !$person-&gt;can('sum'), 'subroutines have been removed from the namespace'; is $person-&gt;add( [qw\/1 3 5 6\/] ), 15, 'Our add() method should work'; throws_ok { My::Names-&gt;new( name =&gt; 'Ovid', created =&gt; 1 ) } 'Moose::Exception', 'Attributes not defined as `param` are illegal in the constructor'; my $doctor = My::Names-&gt;new( name =&gt; 'Smith', title =&gt; 'Dr.' ); is $doctor-&gt;name, 'Dr. Smith', 'Titles should show up correctly'; cmp_ok $doctor-&gt;created, '&gt;=', $person-&gt;created, '... and their created date should be correct'; done_testing;  Obviously, the above code probably won’t be fit for purpose, but it shows you the basics of how you can build an OO system to fit your company’s needs, rather than allowing everyone to just “do their own thing.”  Hire Us  We do code reviews, development, testing, and design, focused on reliability and scalability. Even if you’re just exploring possibilities, feel free to contact us and let’s see what we can do to make your software better.  ","title":"Common Problems in Object-Oriented Code","url":"\/articles\/common-problems-in-object-oriented-code.html"},{"body":"  Originally located at http:\/\/www.pphsg.org\/cdsmith\/types.html , this article explained some basic concepts of type systems. Unfortunately, that page is gone and I had to fetch it from the web archive of that page  . The note at the bottom states that contents are in the public domain. This was published in 2008.  What follows is a short, brilliant introduction to the basic concepts of type systems, by Chris Smith.   What To Know Before Debating Type Systems  I would be willing to place a bet that most computer programmers have, on multiple occasions, expressed an opinion about the desirability of certain kinds of type systems in programming languages. Contrary to popular conception, that’s a great thing! Programmers who care about their tools are the same programmers who care about their work, so I hope the debate rages on.  There are a few common misconceptions, though, that confuse these discussions. This article runs through those I’ve encountered that obscure the most important parts of the debate. My goal is to build on a shared understanding of some of the basic issues, and help people get to the interesting parts more quickly.  Classifying Type Systems  Type systems are commonly classified by several words, of which the most common are “static,” “dynamic,” “strong,” and “weak.” In this section, I address the more common kinds of classification. Some are useful, and some are not.  Strong and Weak Typing  Probably the most common way type systems are classified is “strong” or “weak.” This is unfortunate, since these words have nearly no meaning at all. It is, to a limited extent, possible to compare two languages with very similar type systems, and designate one as having the strong* er * of those two systems. Beyond that, the words mean nothing at all.  Therefore: I give the following general definitions for strong and weak typing, at least when used as absolutes:   Strong typing: A type system that I like and feel comfortable with  Weak typing: A type system that worries me, or makes me feel uncomfortable   What about when the phrase is used in a more limited sense? Then strong typing, depending on the speaker or author, may mean anything on the spectrum from “static” to “sound,” both of which are defined below.  Static and Dynamic Types  This is very nearly the only common classification of type systems that has real meaning. As a matter of fact, its significance is frequently under-estimated. I realize that may sound ridiculous; but this theme will recur throughout this article. Dynamic and static type systems are two completely different things, whose goals happen to partially overlap.  A static type system is a mechanism by which a compiler examines source code and assigns labels (called “types”) to pieces of the syntax, and then uses them to infer something about the program’s behavior. A dynamic type system is a mechanism by which a compiler generates code to keep track of the sort of data (coincidentally, also called its “type”) used by the program. The use of the same word “type” in each of these two systems is, of course, not really entirely coincidental; yet it is best understood as having a sort of weak historical significance. Great confusion results from trying to find a world view in which “type” really means the same thing in both systems. It doesn’t. The better way to approach the issue is to recognize that:   Much of the time, programmers are trying to solve the same problem with static and dynamic types.  Nevertheless, static types are not limited to problems solved by dynamic types.  Nor are dynamic types limited to problems that can be solved with static types.  At their core, these two techniques are not the same thing at all.   Observing the second of these four simple facts is a popular pastime in some circles. Consider this set of presentation notes  , with a rather complicated “the type system found my infinite loop” comment. From a theoretical perspective, preventing infinite loops is in a very deep sense the most basic possible thing you can do with static types! The simply-typed lambda calculus, on which all other type systems are based, proves that programs terminate in a finite amount of time. Indeed, the more interesting question is how to usefully extend the type system to be able to describe programs that don’t terminate! Finding infinite loops, though, is not in the class of things most people associate with “types,” so it’s surprising. It is, indeed, provably impossible with dynamic types (that’s called the halting problem; you’ve probably heard of it!). But it’s nothing special for static types. Why? Because they are an entirely different thing from dynamic types.  The dichotomy between static and dynamic types is somewhat misleading. Most languages, even when they claim to be dynamically typed, have some static typing features. As far as I’m aware, all languages have some dynamic typing features. However, most languages can be characterized as choosing one or the other. Why? Because of the first of the four facts listed above: many of the problems solved by these features overlap, so building in strong versions of both provides little benefit, and significant cost.  Other Distinctions  There are many other ways to classify type systems. These are less common, but here are some of the more interesting ones:   Sound types. A sound type system is one that provides some kind of guarantee. It is a well-defined concept relating to static type systems, and has proof techniques and all those bells and whistles. Many modern type systems are sound; but older languages like C often do not have sound type systems by design; their type systems are just designed to give warnings for common errors. The concept of a sound type system can be imperfectly generalized to dynamic type systems as well, but the exact definition there may vary with usage.  Explicit\/Implicit Types. When these terms are used, they refer to the extent to which a compiler will reason about the static types of parts of a program. All programming languages have some form of reasoning about types. Some have more than others. ML and Haskell have implicit types, in that no (or very few, depending on the language and extensions in use) type declarations are needed. Java and Ada have very explicit types, and one is constantly declaring the types of things. All of the above have (relatively, compared to C and C++, for example) strong static type systems.  The Lambda Cube. Various distinctions between static type systems are summarized with an abstraction called the “lambda cube.” Its definition is beyond the scope of this article, but it basically looks at whether the system provides certain features: parametric types, dependent types, or type operators. Look here  for more information.  Structural\/Nominal Types. This distinction is generally applied to static types with subtyping. Structural typing means a type is assumed whenever it is possible to validly assume it. For example, a record with fields called x, y, and z might be automatically considered a subtype of one with fields x and y. With nominal typing, there would be no such assumed relationship unless it were declared somewhere.  Duck Typing. This is a word that’s become popular recently. It refers to the dynamic type analogue of structural typing. It means that rather than checking a tag to see whether a value has the correct general type to be used in some way, the runtime system merely checks that it supports all of the operations performed on it. Those operations may be implemented differently by different types.   This is but a small sample, but this section is too long already.  Fallacies About Static and Dynamic Types  Many programmers approach the question of whether they prefer static or dynamic types by comparing some languages they know that use both techniques. This is a reasonable approach to most questions of preference. The problem, in this case, is that most programmers have limited experience, and haven’t tried a lot of languages. For context, here, six or seven doesn’t count as “a lot.” On top of that, it requires more than a cursory glance to really see the benefit of these two very different styles of programming. Two interesting consequences of this are:   Many programmers have used very poor statically typed languages.  Many programmers have used dynamically typed languages very poorly.   This section, then, brings up some of the consequences of this limited experience: things many people assume about static or dynamic typing that just ain’t so.  Fallacy: Static types imply type declarations  The thing most obvious about the type systems of Java, C, C++, Pascal, and many other widely-used “industry” languages is not that they are statically typed, but that they are explicitly typed. In other words, they require lots of type declarations. (In the world of less explicitly typed languages, where these declarations are optional, they are often called “type annotations” instead. You may find me using that word.) This gets on a lot of people’s nerves, and programmers often turn away from statically typed languages for this reason.  This has nothing to do with static types. The first statically typed languages were explicitly typed by necessity. However, type inference algorithms - techniques for looking at source code with no type declarations at all, and deciding what the types of its variables are - have existed for many years now. The ML language, which uses it, is among the older languages around today. Haskell, which improves on it, is now about 15 years old. Even C# is now adopting the idea, which will raise a lot of eyebrows (and undoubtedly give rise to claims of its being “weakly typed” -- see definition above). If one does not like type declarations, one is better off describing that accurately as not liking explicit types, rather than static types.  (This is not to say that type declarations are always bad; but in my experience, there are few situations in which I’ve wished to see them required. Type inference is generally a big win.)  Fallacy: Dynamically typed languages are weakly typed  The statement made at the beginning of this thread was that many programmers have used dynamically typed languages poorly. In particular, a lot of programmers coming from C often treat dynamically typed languages in a manner similar to what made sense for C prior to ANSI function prototypes. Specifically, this means adding lots of comments, long variable names, and so forth to obsessively track the “type” information of variables and functions.  Doing this prevents a programmer from realizing the benefits of dynamic typing. It’s like buying a new car, but refusing to drive any faster than a bicycle. The car is horrible; you can’t get up the mountain trails, and it requires gasoline on top of everything else. Indeed, a car is a pretty lousy excuse for a bicycle! Similarly, dynamically typed languages are pretty lousy excuses for statically typed languages.  The trick is to compare dynamically typed languages when used in ways that fit in with their design and goals. Dynamically typed languages have all sorts of mechanisms to fail immediately and clearly if there is a runtime error, with diagnostics that show you exactly how it happened. If you program with the same level of paranoia appropriate to C - where a simple bug may cause a day of debugging - you will find that it’s tough, and you won’t be actually using your tools.  (As a side comment, and certainly a more controversial one, the converse is equally true; it doesn’t make sense to do the same kinds of exhaustive unit testing in Haskell as you’d do in Ruby or Smalltalk. It’s a waste of time. It’s interesting to note that the whole TDD movement comes from people who are working in dynamically typed languages... I’m not saying that unit testing is a bad idea with static types; only that it’s entirely appropriate to scale it back a little.)  Fallacy: Static types imply upfront design or waterfall methods  Some statically typed languages are also designed to enforce someone’s idea of a good development process. Specifically, they often require or encourage that you specify the whole interface to something in one place, and then go write the code. This can be annoying if one is writing code that evolves over time or trying out ideas. It sometimes means changing things in several different places in order to make one tweak. The worst form of this I’m aware of (though done mainly for pragmatic reasons rather than ideological ones) is C and C++ header files. Pascal has similar aims, and requires that all variables for a procedure or function be declared in one section at the top. Though few other languages enforce this separation in quite the same way or make it so hard to avoid, many do encourage it.  It is absolutely true that these language restrictions can get in the way of software development practices that are rapidly gaining acceptance, including agile methodologies. It’s also true that they have nothing to do with static typing. There is nothing in the core ideas of static type systems that has anything to do with separating interface from implementation, declaring all variables in advance, or any of these other organizational restrictions. They are sometimes carry-overs from times when it was considered normal for programmers to cater to the needs of their compilers. They are sometimes ideologically based decisions. They are not static types.  If one doesn’t want a language deciding how they should go about designing their code, it would be clearer to say so. Expressing this as a dislike for static typing confuses the issue.  This fallacy is often stated in different terms: “I like to do exploratory programming” is the popular phrase. The idea is that since everyone knows statically typed languages make you do everything up front, they aren’t as good for trying out some code and seeing what it’s like. Common tools for exploratory programming include the REPL (read-eval-print loop), which is basically an interpreter that accepts statements in the language a line at a time, evaluates them, and tells you the result. These tools are quite useful, and they exist for many languages, both statically and dynamically typed. They don’t exist (or at least are not widely used) for Java, C, or C++, which perpetuates the unfortunate myth that they only work in dynamically typed languages. There may be advantages for dynamic typing in exploratory programming (in fact, there certainly are some advantages, anyway), but it’s up to someone to explain what they are, rather than just to imply the lack of appropriate tools or language organization.  Fallacy: Dynamically typed languages provide no way to find bugs  A common argument leveled at dynamically typed languages is that failures will occur for the customer, rather than the developer. The problem with this argument is that it very rarely occurs in reality, so it’s not very convincing. Programs written in dynamically typed languages don’t have far higher defect rates than programs written in languages like C++ and Java.  One can debate the reasons for this, and there are good arguments to be had there. One reason is that the average skill level of programmers who know Ruby is higher than those who know Java, for example. One reason is that C++ and Java have relatively poor static type systems. Another reason, though, is testing. As mentioned in the aside above, the whole unit testing movement basically came out of dynamically typed languages. It has some definite disadvantages over the guarantees provided by static types, but it also has some advantages; static type systems can’t check nearly as many properties of code as testing can. Ignoring this fact when talking to someone who really knows Ruby will basically get you ignored in turn.  Fallacy: Static types imply longer code  This fallacy is closely associated with the one above about type declarations. Type declarations are the reason many people associated static types with a lot of code. However, there’s another side to this. Static types often allow one to write much more concise code!  This may seem like a surprising claim, but there’s a good reason. Types carry information, and that information can be used to resolve things later on and prevent programmers from needing to write duplicate code. This doesn’t show up often in simple examples, but a really excellent case is found in the Haskell standard library’s Data.Map module. This module implements a balanced binary search tree, and it contains a function whose type signature looks like this:  lookup :: (Monad m, Ord k) =&gt; k -&gt; Map k a -&gt; m a  This is a magical function. It says that I can look something up in a Map and get back the result. Simple enough, but here’s the trick: what do I do if the result isn’t there? Common answers might include returning a special “nothing” value, or aborting the current computation and going to an error handler, or even terminating the whole program. The function above does any of the above! Here’s how I compare the result against a special nothing value:  case (lookup bobBarker employees) of Nothing -&gt; hire bobBarker Just salary -&gt; pay bobBarker salary  How does Haskell know that I want to choose the option of getting back Nothing when the value doesn’t exist, rather than raising some other kind of error? It’s because I wrote code afterward to compare the result against Nothing ! If I had written code that didn’t immediately handle the problem but was called from somewhere that handled errors three levels up the stack, then lookup would have failed that way instead, and I’d be able to write seven or eight consecutive lookup statements and compute something with the results without having to check for Nothing all the time. This completely dodges the very serious “exception versus return value” debate in handling failures in many other languages. This debate has no answer. Return values are great if you want to check them now; exceptions are great if you want to handle them several levels up. This code simply goes along with whatever you write the code to do.  The details of this example are specific to Haskell, but similar examples can be constructed in many statically typed languages. There is no evidence that code in ML or Haskell is any longer than equivalent code in Python or Ruby. This is a good thing to remember before stating, as if it were obviously true, that statically typed languages require more code. It’s not obvious, and I doubt if it’s true.  Benefits of Static Types  My experience is that the biggest problems in the static\/dynamic typing debate occur in failing to understand the issues and potential of static types. The next two sections, then, are devoted to explaining this position in detail. This section works upward from the pragmatic perspective, while the next develops it into its full form.  There are a number of commonly cited advantages for static typing. I am going to list them in order from least to most significant. (This helps the general structure of working up to the important stuff.)  Performance  Performance is the gigantic red herring of all type system debates. The knowledge of the compiler in a statically typed language can be used in a number of ways, and improving performance is one of them. It’s one of the least important, though, and one of the least interesting.  For most computing environments, performance is the problem of two decades ago. Last decade’s problem was already different, and this decade’s problems are at least 20 years advanced beyond performance being the main driver of technology decisions. We have new problems, and performance is not the place to waste time.  (On the other hand, there are a few environments where performance still matters. Languages in use there are rarely dynamically typed, but I’m not interested enough in them to care much. If you do, maybe this is your corner of the type system debate.)  Documentation  If, indeed, performance is irrelevant, what does one look to next? One answer is documentation. Documentation is an important aspect of software, and static typing can help.  Why? Because documentation isn’t just about comments. It’s about everything that helps people understand software. Static type systems build ideas that help explain a system and what it does. They capture information about the inputs and outputs of various functions and modules. This is exactly the set of information needed in documentation. Clearly, if all of this information is written in comments, there is a pretty good chance it will eventually become out of date. If this information is written in identifier names, it will be nearly impossible to fit it all in. It turns out that type information is a very nice place to keep this information.  That’s the boring view. As everyone knows, though, it’s better to have self-documenting code than code that needs a lot of comments (even if it has them!). Conveniently enough, most languages with interesting static type systems have type inference, which is directly analogous to self-documenting code. Information about the correct way to use a piece of code is extracted from the code itself (i.e., it’s self-documenting), but then verified and presented in a convenient format. It’s documentation that doesn’t need to be maintained or even written, but is available on demand even without reading the source code.  Tools and Analysis  Things get way more interesting than documentation, though. Documentation is writing for human beings, who are actually pretty good at understanding code anyway. It’s great that the static type system can help, but it doesn’t do anything fundamentally new.  Fundamentally new things happen when type systems help computer programs to understand code. Perhaps I need to explain myself here. After all, a wise man (Martin Fowler, IIRC) one said:   “Any fool can write code that a computer can understand. Good programmers write code that humans can understand.”   I don’t disagree with Martin Fowler, but we have different definitions of understand in mind. Getting a computer to follow code step by step is easy. Getting a computer to analyze it and answer more complex questions about it is a different thing entirely, and it is very hard.  We often want our development tools to understand code. This is a big deal. I’ll turn back to Martin Fowler, who points this out as well  .  Correctness  Ultimately, though, the justification for static typing has to come back to writing correct code. Correctness, of course, is just the program doing “what you want.”  This is a really tough problem; perhaps the toughest of all. The theory of computation has a result called Rice’s Theorem, which essentially says this: Given an arbitrary program written in a general purpose programming language, it is impossible to write a computer program that determines anything about the program’s output. If I’m teaching an intro to programming class and assign my students to write “hello world”, I can’t program a grader to determine if they did so or not. There will be some programs for which the answer is easy; if the program never makes any I\/O calls, then the answer is no. If the program consists of a single print statement, it’s easy to check if the answer is yes. However, there will be some complicated programs for which my grader can never figure out the answer. (A minor but important technical detail: one can’t run the program and wait for it to finish, because the program might never finish!) This is true of any statement about programs, including some more interesting ones like “does this program ever finish?” or “does this program violate my security rules?”  Given that we can’t actually check the correctness of a program, there are two approaches that help us make approximations:   Testing: establishes upper bounds on correctness  Proof: establishes lower bounds on correctness   Of course, we care far more about lower bounds than upper bounds. The problem with proofs, though, is the same as the problem with documentation. Proving correctness is easy only somewhat insanely difficult when you have a static body of code to prove things about. When the code is being maintained by three programmers and changing seven times per day, maintaining the correctness proofs falls behind. Static typing here plays exactly the same role as it does with documentation. If (and this is a big if) you can get your proofs of correctness to follow a certain form that can be reproduced by machine, the computer itself can be the prover, and let you know if the change you just made breaks the proof of correctness. The “certain form” is called structural induction (over the syntax of the code), and the prover is called a type checker.  An important point here is that static typing does not preclude proving correctness in the traditional way, nor testing the program. It is a technique to handle those cases in which testing might be guaranteed to succeed so they don’t need testing; and similarly, to provide a basis from which the effort of manual proof can be saved for those truly challenging areas in which it is necessary.  Dynamic Typing Returns  Certainly dynamic typing has answers to this. Dynamically typed languages can sometimes perform rather well (see Dylan), sometimes have great tools (see Smalltalk), and I’m sure they occasionally have good documentation as well, though the hunt for an example is too much for me right now. These are not knock-down arguments for static typing, but they are worth being aware of.  The correctness case is particularly enlightening. Just as static types strengthened our proofs of correctness by making them easier and automatic, dynamic typing improves testing by making it easier and more effective. It simply makes the code fail more spectacularly. I find it amusing when novice programmers believe their main job is preventing programs from crashing. I imagine this spectacular failure argument wouldn’t be so appealing to such a programmer. More experienced programmers realize that correct code is great, code that crashes could use improvement, but incorrect code that doesn’t crash is a horrible nightmare.  It is through testing, then, that dynamically typed languages establish correctness. Recall that testing establishes only upper bounds on correctness. (Dijkstra said it best: “Program testing can be used to show the presence of bugs, but never to show their absence.“) The hope is that if one tries hard enough and still fails to show the presence of bugs, then their absence becomes more likely. If one can’t seem to prove any better upper bound, then perhaps the correctness really is 100%. Indeed, there is probably at some correlation in that direction.  What is a Type?  This is as good a point as any to step back and ask the fundamental question: what is a type? I’ve already mentioned that I think there are two answers. One answer is for static types, and the other is for dynamic types. I am considering the question for static types.  It is dangerous to answer this question too quickly. It is dangerous because we risk excluding some things as types, and missing their “type” nature because we never look for it. Indeed, the definition of a type that I will eventually give is extremely broad.  Problems with Common Definitions  One common saying, quoted often in an attempt to reconcile static and dynamic typing, goes something like this: Statically typed languages assign types to variables, while dynamically typed languages assign types to values. Of course, this doesn’t actually define types, but it is already clearly and obviously wrong. One could fix it, to some extent, by saying “statically typed languages assign types to expressions, ...” Even so, the implication that these types are fundamentally the same thing as the dynamic version is quite misleading.  What is a type, then? When a typical programmer is asked that question, they may have several answers. Perhaps a type is just a set of possible values. Perhaps it is a set of operations (a very structural-type-ish view, to be sure). There could be arguments in favor of each of these. One might make a list: integers, real numbers, dates, times, and strings, and so on. Ultimately, though, the problem is that these are all symptoms rather than definitions. Why is a type a set of values? It’s because one of the things we want to prove about our program is that it doesn’t calculate the square roots of a string. Why is a type a set of operations? It’s because one of the things we want to know is whether our program attempts to perform an impossible operation.  Let’s take a look at another thing we often want to know: does our web application stick data from the client into SQL queries without escaping special characters first? If this is what we want to know, then these become types. This article by Tom Moertel  builds this on top of Haskell’s type system. So far, it looks like a valid definition of “type” is as follows: something we want to know.  A Type System  Clearly that’s not a satisfactory definition of a type. There are plenty of things we want to know that types can’t tell us. We want to know whether our program is correct, but I already said that types provide conservative lower bounds on correctness, and don’t eliminate the need for testing or manual proof. What makes a type a type, then? The other missing component is that a type is part of a type system.  Benjamin Pierce’s book Types and Programming Languages  is far and away the best place to read up on the nitty gritty details of static type systems, at least if you are academically inclined. I’ll quote his definition.   A type system is a tractable syntactic method for proving the absence of certain program behaviors by classifying phrases according to the kinds of values they compute.   This is a complex definition, but the key ideas are as follows:   syntactic method .. by classifying phrases : A type system is necessarily tied to the syntax of the language. It is a set of rules for working bottom up from small to large phrases of the language until you reach the result.  proving the absence of certain program behaviors : This is the goal. There is no list of “certain” behaviors, though. The word just means that for any specific type system, there will be a list of things that it proves. What it proves is left wide open. (Later on in the text: “... each type system comes with a definition of the behaviors it aims to prevent.“)  tractable : This just means that the type system finishes in a reasonable period of time. Without wanting to put words in anyone’s mouth, I think it’s safe to say most people would agree that it’s a mistake to include this in the definition of a type system. Some languages even have undecidable type systems. Nevertheless, it is certainly a common goal; one doesn’t expect the compiler to take two years to type-check a program, even if the program will run for two years.   The remainder of the definition is mainly unimportant. The “kinds of values they compute” is basically meaningless unless we know what kinds we might choose from, and the answer is any kind at all.  An example looks something like this. Given the expression 5 + 3 , a type checker may look at 5 and infer that it’s an integer. It may look at 3 and infer it’s an integer. It may then look at the + operator, and know that when + is applied to two integers, the result is an integer. Thus it’s proven the absence of program behaviors (such as adding an integer to a string) by working up from the basic elements of program syntax.  Examples of Unusual Type Systems  That was a pretty boring example, and one that plays right into a trap: thinking of “type” as meaning the same thing it does in a dynamic type system. Here are some more interesting problems being solved with static types.   http:\/\/wiki.di.uminho.pt\/twiki\/pub\/Personal\/Xana\/WebHome\/report.pdf  . Uses types to ensure that the correct kinds of data are gotten out of a relational database. Via the type system, the compiler ends up understanding how to work with concepts like functional dependencies and normal forms, and can statically prove levels of normalization.  http:\/\/www.cs.bu.edu\/~hwxi\/academic\/papers\/pldi98.pdf  . Uses an extension to ML’s type system to prove that arrays are never accessed out of bounds. This is an unusually hard problem to solve without making the languages that solve it unusable, but it’s a popular one to work on.  http:\/\/www.cis.upenn.edu\/~stevez\/papers\/LZ06a.pdf  . This is great. This example uses Haskell’s type system to let someone define a security policy for a Haskell program, in Haskell, and then proves that the program properly implements that security. If a programmer gets security wrong, the compiler will complain rather than opening up a potential security bug in the system.  http:\/\/www.brics.dk\/RS\/01\/16\/BRICS-RS-01-16.pdf  . Just in case you thought type systems only solved easy problems, this bit of Haskell gets the type system to prove two central theorems about the simply typed lambda calculus, a branch of computation theory!   The point of these examples is to point out that type systems can solve all sorts of programming problems. For each of these type systems, concepts of types are created that represent the ideas needed to accomplish this particular task with the type system. Some problems solved by static type systems look nothing like the intuitive idea of a type. A buggy security check isn’t normally considered a type error, but only because not many people use languages with type systems that solve that problem.  To reiterate the point above, it’s important to understand how limiting it is to insist, as many people do, that the dynamic typing definition of a “type” is applied to static typing as well. One would miss the chance to solve several real-world problems mentioned above.  The True Meaning of Type  So what is a type? The only true definition is this: a type is a label used by a type system to prove some property of the program’s behavior. If the type checker can assign types to the whole program, then it succeeds in its proof; otherwise it fails and points out why it failed. This is a definition, then, but it doesn’t tell us anything of fundamental importance. Some further exploration leads us to insight about the fundamental trade-offs involved in using a static type checker.  If you were looking at things the right way, your ears may have perked up a few sections back, when I said that Rice’s Theorem says we can’t determine anything about the output of a program. Static type systems prove properties of code, but it almost appears that Rice’s Theorem means we can’t prove anything of interest with a computer. If true, that would be an ironclad argument against static type systems. Of course, it’s not true. However, it is very nearly true. What Rice’s Theorem says is that we can’t determine anything. (Often the word “decide” is used; they mean the same thing here.) It didn’t say we can’t prove anything. It’s an important distinction!  What this distinction means is that a static type system is a conservative estimate. If it accepts a program, then we know the program has the properties proven by that type checker. If it fails... then we don’t know anything. Possibly the program doesn’t have that property, or possibly the type checker just doesn’t know how to prove it. Furthermore, there is an ironclad mathematical proof that a type checker of any interest at all is always conservative. Building a type checker that doesn’t reject any correct programs isn’t just difficult; it’s impossible.  That, then, is the trade-off. We get assurance that the program is correct (in the properties checked by this type checker), but in turn we must reject some interesting programs. To continue the pattern, this is the diametric opposite of testing. With testing, we are assured that we’ll never fail a correct program. The trade-off is that for any program with an infinite number of possible inputs (in other words, any interesting program), a test suite may still accept programs that are not correct - even in just those properties that are tested.  Framing the Interesting Debate  That last paragraph summarizes the interesting part of the debate between static and dynamic typing. The battleground on which this is fought out is framed by eight questions, four for each side:   For what interesting properties of programs can we build static type systems?  How close can we bring those type systems to the unattainable ideal of never rejecting a correct program?  How easy can it be made to program in a language with such a static type system?  What is the cost associated with accidentally rejecting a correct computer program?  For what interesting properties of programs can we build test suites via dynamic typing?  How close can we bring those test suites to the unattainable ideal of never accepting a broken program?  How easy can it be made to write and execute test suites for programs?  What is the cost associated with accidentally accepting an incorrect computer program?   If you knew the answer to those eight questions, you could tell us all, once and for all, where and how we ought to use static and dynamic typing for our programming tasks.  ","title":"What to Know Before Debating Type Systems","url":"\/articles\/what-to-know-before-debating-type-systems.html"},{"body":"    History  Variables and Types  Classes are Types  The Problem with :builder  Builder Guidelines     I’ve been spending a lot of time designing a new OO syntax for the Perl language  called Corinna. This syntax is not designed to be another object-oriented module. Instead, it’s designed to be a be a core part of the language. It’s received a lot of support (and a few detractors), but I have a problem with it.  Lately I’ve been worried about the slot variable :builder attribute and I’ve been concerned that it’s problematic. I’ve nattered on about this on IRC and just generally let the concerns simmer in my mind. If we’re going to get to propose something for the core, it’s better that we avoid mistakes before that time rather than after. Then Damian Conway shared some thoughts with me about his concerns for :builder and that was the final straw. Something has to change.   History  As part of our proposal, we have this:  class SomeClass { has $x :builder; method _build_x () { return rand; } }  If you call SomeClass-&gt;new , you’ll get a random numeric value assigned to the $x slot variable. Note that $x has no reader or writer, so it’s designed to be completely encapsulated. But it’s not and that’s a problem.  That’s been bugging me for quite a while, but to explain that, I need to go back in time.  In 1967 (my birth year, but I swear it’s a coincidence), the first truly class-based OO language was released, Simula 67  and it introduced classes, polymorphism, encapsulation, and inheritance (and coroutines (!), but let’s not go there).  Classes, polymorphism, and encapsulation are relatively uncontroversial, but the debate over inheritance has raged for decades.  Alan Kay, the man who invented the term “Object Oriented” in Utah “sometime after 1966”  and who’s been doing OO programming before most of us were born, has made it clear over the years that inheritance in OO is very problematic. He’s even left it out of the implementation of some OO languages he’s created. And then I found a Quora question entitled “What does Alan Kay think about inheritance in object-oriented programming?” and someone purporting to be Alan Kay responded  with a deeply thoughtful answer. His first paragraph is fantastic (emphasis mine):   Simula I didn’t have inheritance (paper ca 1966) and Simula 67 did (paper ca 1968 or so). I initially liked the idea — it could be useful — but soon realized that something that would be “mathematically binding” was really needed because the mechanism itself let too many semantically different things to be “done” (aka “kluged”) by the programmer. For example, there is no restriction of any kind to have a subclass resemble a superclass, be a refinement of a superclass, etc. All relies on the cleanliness of mind of programmers (and even the most clean of these often just do things they need when in the throes of debugging).   In other words, if I create a class named Vehicle::Motorized , there is nothing to stop your Order::Customer class from inheriting from it. Would that make sense? Probably not. But there we are.  Next, let’s take a look at a Python object:  class Point: def __init__(self, x, y): self.x = x self.y = y def inverse(self): return Point(self.y,self.x) point = Point(7,3.2).inverse() print(point.x) point.x = &quot;foo&quot;  And for those who would like to see that in our current proposal:  class Point { has ($x,$y) :reader :writer :new; method inverse { return $class-&gt;new( x =&gt; $y, y =&gt; $x ); } } my $point = Point-&gt;new( x =&gt; y =&gt; 3.2 )-&gt;inverse; say $point-&gt;x; $point-&gt;x(&quot;foo&quot;);  As you can see, we’ve assigned “foo” to the x value of a Point object. We’ve done this in both Python and our new OO syntax for Perl. Sadly, due to the need to harmonize types across the language, we can’t have type constraints in v0.1.0. This is a serious limitation, but we’re taking baby steps.  But what about the Python version? Surely they must validate their data, right? No. Pythonistas have assured me that validating your data is very unpythonic  .  (I envision spit-takes from tons of Moo\/se fans out there right now).  Python 3.7 contains data classes  and those will help, but I don’t know how widespread the usage is.   Variables and Types  But what does that have to do with inheritance?  For many languages, you can try something like (example in the Go programming language) var myvar float64 = \"foo\" and it will throw an exception. It might be compile-time. It might be runtime. But it will die a horrible death because the variable, myvar , has a type associated with it and if you assign something that doesn’t match that type (or that the language can’t\/won’t coerce), then things go boom.  Why is that? Because on line 372 of your whopping huge function , it’s easy to have something like myvar := stuff[0:2] and not be sure what kind of data stuff[0:2] contains.  So we can say that—for a small subset of types the programming language can understand—variables in those kinds of languages are the “experts” about what they can do and if you try to put that variable in an invalid state, it blows up.  That’s great. But what about var celsius float64 = -1300.15 . Is that valid?  Absolute zero is -273.15 on the Celsius scale, so we can make a reasonable argument that the above is not valid. In languages like Raku  , we easily fix that:  subset Celsius of Num where * &gt;= -273.15; my Celsius $temp = -1300.15;  The above will fail at compile-time. Most languages that allow types, however, aren’t quite as flexible, but you can use classes instead.  Classes are Types  my $temp = Celsius-&gt;new( temperature =&gt; -1300.15 );  Assuming your class is sane, the above should throw an exception because you’re trying to create an object with an invalid state.  If you think that’s similar to how built-in types work, it is. We can think of a class as a user-defined type. Types have a set of allowed values and operators you can apply to those values. Classes have (often complex) sets of allowed values and methods (operators) you can apply to those values.  Just as an int is often an expert in values -32,768 to 32,767, or -2,147,483,648 to 2,147,483,647 if you’re using 4 bytes (larger values for more bytes, of course), so is your Celsius class an expert about allowed Celsius temperatures. And your Personal Shopper knows exactly how much money it can spend, and even if it knows you have enough money to buy cocaine, it’s still going to throw an exception if you request that because it’s an illegal operation.    The Problem with :builder  And this gets us to :builder . In Perl‘s Moo\/se family of OO, you can do something similar to this (but using our new syntax):  class Parent { has $x :builder; method _build_x () { return rand; } } class Child isa Parent { method _build_x () { return &quot;not a number&quot; } }  This has exactly the same issue with see with Python’s point.x = \"foo\" . It’s hardly necessary to belabor the point, but I will: $x is supposed to be private, but we’ve grossly violated encapsulation.  Ah, you reply. But Moo\/se has types. So let’s look at Moo\/se, using a better example (not great, but enough to get the point across):  package Collection { use Moose; # this is a personal module which gives me a &quot;saner&quot; Perl # environment use Less::Boilerplate; use Types::Standard qw(Int ArrayRef); has _index =&gt; ( is =&gt; 'rw', isa =&gt; Int-&gt;where('$_ &gt;= 0'), builder =&gt; '_build__index', ); # default, but maybe someone wants a different default sub _build__index { 0 } has items =&gt; ( is =&gt; 'ro', isa =&gt; ArrayRef, required =&gt; 1, ); sub BUILD ( $self, @ ) { my @items = $self-&gt;items-&gt;@*; my $type = ref $items[0]; foreach my $item (@items) { if ( not defined $item ) { croak(&quot;items() does not allow undefined values&quot;); } if ( ref $item ne $type ) { croak(&quot;All items in collection must be of the same type&quot;); } } } sub num_items ($self) { return scalar $self-&gt;items-&gt;@*; } sub next ($self) { my $i = $self-&gt;_index; return if $i &gt;= $self-&gt;num_items; $self-&gt;_index( $i + 1 ); return $self-&gt;items-&gt;[$i]; } sub reset ($self) { $self-&gt;_index(0); } }  It goes without saying that writing an iterator like the above can be rather problematic for a number of reasons, and yes, the _build__index method is kind of silly, but it shows the issue. For example, what would a negative index do? In the above, it would throw an exception because of the Int-&gt;where( '$_ &gt;= 0' ) constraint. But what would happen if we set the index to a value larger than the size of the items array reference (and ignoring that the above allows $coll-&gt;_index($integer) ).  package My::Collection { use Moose; extends 'Collection'; sub _build__index { 5 } } my $collection = My::Collection-&gt;new( items =&gt; [qw\/foo bar baz\/] ); while ( defined( my $item = $collection-&gt;next ) ) { say $item; }  What do you think the above does? The constraint passes, but it prints nothing. That’s because the allowed value of the index is tightly coupled to something else in that class, the number of items in the collection.  Oh, but Ovid, I wouldn’t write something like that!  Yes, you would. I do it too. We all do it when we’re quickly hacking on that 883 line class from our 1,000 KLOC codebase and we’re rushing to beat our sprint deadline. Programmers do this all the time because we have a language which allows this, but OO modules which encourage this. You can inspect tons of OO modules and find attributes (a.k.a “slot variables” in our new system) which are coupled with other values. We like to minimize that, but sometimes we can’t and classes are supposed to encapsulate this problem and handle it internally.  My suggestion is that, if you want to allow this internally, you have to do it manually with the ADJUST phaser  .  class Collection { has $index; has $items :new; ADJUST (%args) { # other checks here $index = $self-&gt;_default_index; if ( $index &gt; $items-&gt;@* - 1 ) { # don't let subclasses break our code croak(...); } } method _default_index () { 0 } # other methods here }  Do I recommend that? No. Can you do it? Yes. And it has some advantages.  First, you are being explicit about which slots are can access in your subclasses. Second, you have fine-grained control over slot initialization ordering.  By default, with Corinna, slots are initialized in the order declared (until you hit :new , in which case all :new slots are initialized at once). In Moo\/se it’s harder to predict the order of initialization. That’s why you often see a bunch of attributes declared as lazy to ensure that non-lazy attributes are set first. If you have circular lazy dependencies, it can be hard to work out and you fall back to BUILD or BUILDARGS as needed.  Yes, this does mean a touch more work on the off chance you need to override a slot in a subclass. I’ve been writing Moo\/se for years and I can count on the fingers of one hand the number of times I’ve needed to do this. However, I can’t count how many times I’ve unnecesssarily littered my namespace with sub _build... methods because this is largely what Moo\/se prefers and what developers will call you out on in code reviews if you do it some other way.  Oh, and we also hit issues like this. Which of these is correct?  class Position { slot $x :name(ordinate) :builder; # and many lines later... method _build_ordinate { return rand } # Does the builder follow the public slot name? } class Position { slot $x :name(ordinate) :builder; # and many lines later... method _build_x { return rand } # Or does it follow the private slot variable name? }  If we were to allow builders, we should, at least, mandate the syntax :builder($method_name) and not allow default builder names.  Builder Guidelines  So here are a few guidelines we should follow for assigning default values to slots.   Every slot should have a default value.   It is often a code smell to have an undefined slot.   Use the = default syntax   In the example below, :new says we can pass the value to the constructor, but if we don’t, it will get the value 42.  has $x :new = 42;   YAGNI (You Ain’t Gonna Need It).   Don’t allow overriding a private slot value unless you have a clear need for it. This also prevents littering the namespace with a _build_... methods.   Liskov is your friend   Remember the Liskov Substitution Principle  : if you have a subclass and it must be allowed to override some of this data, remember that a subclass should be a more specialized version of a parent class and must be usable anywhere the parent can be.   Check for coupling   If the slot data is tightly coupled to some other slot data, consider breaking the coupling or ask yourself if delegation is more appropriate. Have checks in BUILD (Moo\/se) or ADJUST (Corinna) to verify the integrity of your class.  If all of the above seems like too much to remember, just don’t allow child classes to set parent slot data. Corinna should follow one of the early principles guiding its design: you should be allowed to do bad things, but the language design shouldn’t encourage bad things.  ","title":"Classes Should Not Override Parent Attributes","url":"\/articles\/the-problem-with-builder.html"},{"body":" It's safe to say that out of all of the software development project problems I've come across, there's one I don't know how to solve: the tyranny of budgets. If you can provide any insight, please leave a comment below. I'd love to hear it.  A Typical Horror Story  This following is a story I've told a few times, so bear with me if you've heard it before.  A large European company was earning millions of euros (a low seven figures) from their core software and was considered a \"market leader\" in what they do. Their software was written in Perl, long ago, by developers who didn't understand Perl or software development. Those software developers were long gone. Thus, after well over a decade of \"how can we squeeze this in\" development, they had a steaming pile of ones and zeros that no self-respecting software developer would want to work on.  Fortunately, I am no self-respecting software developer. I like rebuilding legacy systems the same way my friend James loves rebuilding old cars. Except that the company didn't call me to rebuild their software. They decided that this highly profitable software was going to be thrown away and rewritten in Java. They could find more developers and the code would be \"cleaner.\" Except, as many companies discover far too late, rewrites are often financial disasters that drag on forever . The old code base had a ton of features that were time-consuming to port over and by the time they called me to \"maintain\" the Perl code, a couple of years had gone by and the company only had two customers, out of hundreds, using the new system.  Given that I have a lot of experience working with legacy codebases, the initial conversations went well. I clearly understood their problem domain, I am a deep expert in this area, and I was able to identify several of their issues before I had even seen the code   They were sold; I was the person who was going to help stabilize the code base earning them millions every year.  And then the trainwreck happened.  They wanted to have me part-time, on call, and pay me a junior developer's rates.  Excuse me? You're unable to maintain your core product and you're looking for a part-time junior dev for this?  \"We're phasing this product out and that's all we have in our budget.\"    A year later, they called me back. It seems the $100\/day developer they found wasn't that good of a deal after all and he eventually stopped returning their phone calls. They still didn't have the new system ready and estimated it would take at least a year before it could properly launch. Would I be interested in working with them on the legacy system?  Of course I would. This happens all the time in consulting and you either accept it or go back to being an employee.  Then they explained they wanted to have me part-time, on call, and pay me a junior developer's rates.  Fast forward to today, several years later. I check their web site and see they're still in business, but curiously, no mention is made of the core business they were in. They seem to have shifted into a related field, cunningly using the word \"AI\" in its description.   . Maybe it was a brilliant business decision? Maybe it was a Hail Mary pass  ? Maybe it was a gradual evolution? Who knows? But I suspect they didn't throw away their multi-million euro core product on a whim.  Used Cars  I've always been amazed at how many companies treat their software like a college student buying a used car. They have a limited budget. They buy that old Ford Mustang for $1,500. They complain bitterly at their maintenance costs. And their fuel costs. And their insurance costs!   And the time missed from school\/work dealing with said costs.  In fact, I was the one who bought that old Mustang and it was a beast. Its fuel consumption should have been measured in gallons per mile, it pissed oil all over the place and, thanks to an exhaust leak from a warped manifold, it kept burning through the wires to the starter, causing me to replace the starter every few months.   That $1,500 was all I thought I could afford but it cost me a hell of a lot more than a more expensive car.  But you know what? I didn't have much of a choice. Local public transportation was a crap shoot, meaning I risked missing work and losing my job. The distance between home, work, and college meant I couldn't walk or ride a bike and I could not afford to move. It was a cheap car or drop out of school. It was all I could afford and I couldn't afford it.  Software Projects  But there is a critical difference between used cars and software projects.   . It's easy to get another car, if you have the money. The same isn't true for custom software. Maybe you get a great deal on that used car or maybe it breaks down every few months and has to be repaired. Maybe the engine block cracks and you have to junk the car. Do you really want to roll those dice on your software?  The answer, for many companies, is \"yes.\" More than once I've heard decision makers say \"this makes perfect sense and would save us a lot of money in the long run, but in the short run, we don't have the budget for it.\"  And you know what? I get that. While it's often penny wise and pound foolish, sometimes companies just don't have the money. Or the project isn't that important to the company.   . Or, as in the case of the company that kept trying to hire me part-time, sometimes it's simply inertia: \"we've made a decision and now we don't have to think about it.\"  And the last reason is why so much of our software is rubbish.  I honestly don't know how to break a company out of that thinking, so honestly, we don't even try. We politely explain our point of view, and move on to clients who understand that buying a cheap used car is a crap shoot. But I wish I could figure out how to get more people to read Douglas Hubbard and learn that development costs are the least important aspect of estimating project value .  Today, there's a Volkswagen Golf in front of our house. It cost more to buy than that Mustang did, but it costs far less per month and will do so for a long, long time. It was a good choice.  ","title":"The Tyranny of Budgets","url":"\/articles\/the-tyranny-of-budgets.html"},{"body":" Note: the following details a serious, expensive class of bugs that your project may have, but that many don't know about. Out of necessity, I'll lightly touch on technical details.  I was hiring developers for a large, complicated application. I happened to mention to one of the more promising candidates that the application was fairly write-heavy and we might experience some performance concerns later. This developer, with years of experience and skills across multiple technologies that might have been of use to us, replied \"you're going to either need to shard your database or switch to NoSQL.\"  That was enough to guarantee they didn't get the job.  It's a distressingly common problem: with absolutely no understanding of how the database was designed or what the application did, an experienced developer made a strong declaration about how to fix a problem we didn't yet have. In fact, when companies hire me for consulting, one of the first things I do is examine the data storage layer. It's often in pretty bad shape and a constant source of bugs because developers learn SQL and think they understand databases.  If you take nothing else from this article, take this: learning how to write and optimize SQL has as much in common with designing a database as being an expert mechanic qualifies you to design a car .  Data Quality Woes  In many startups, there's not a lot of up front money, so a qualified DBA is viewed as a luxury rather than a necessity. As a result, the devs choose a data storage tool based on what they're comfortable with instead of what the project actually requires. MySQL is so popular and easy to use that many developers choose it out of habit, tune a few parameters for performance, and start developing.  As the project grows, data issues are noticed and get fixed on an ad hoc basis. Variants of the following bug abound:  mysql&gt; create table example (tooshort varchar(3)); Query OK, 0 rows affected (0.00 sec) mysql> insert into example (tooshort) values (\"12345\"); Query OK, 1 row affected, 1 warning (0.00 sec) mysql&gt; select tooshort from example; +----------+ | tooshort | +----------+ | 123 | +----------+  As you can see from the above, MySQL happily truncated the data, throwing away valuable information. It issued a warning, but most applications ignore database warnings. For one of our clients, we found two years of analytic data had to be thrown away because of this bug, even though this warning was issued on every insert.  Once developers learn about MySQL's strict mode, they often try to enable it, only to watch the application crash and burn because it was written with the assumption that division by zero produces NULL values instead of errors, \"0000-00-00\" is a real date, and many other tiny problems. These same applications usually lack foreign key constraints, or use them inconsistently. And more than one developer has solved deadlock issues by not using database transactions.  By the time strict mode is finally enabled, there's a layer of bugs in the database and a layer of related \"strict mode workarounds\" in the application level. It becomes expensive to maintain and much development effort is spent on correcting existing problems rather than build new features that could earn new business.  Those who use Oracle or SQL Server often have databases just as bad, so this isn't just a \"MySQL thing.\" Curiously, we've noticed that our clients who use PostgreSQL tend to have better quality databases, possibly stemming from a database community which is obsessed with data quality.  Performance Woes  For one company, a domain name registrar, I was helping them work on fraud prevention tools when a project manager mentioned that a particular query took 15 minutes to return. The team lead explained that there was \"too much data\" to query efficiently. That \"too much data\" was only three million records. After running explain on the query, I discovered they weren't using indexes. A few minutes later, that 15 minute query took under two seconds to run. Were it not for the accident of my being there, they would have concluded the database was \"too slow\" and reached for a NoSQL solution.  Another common issue, endemic with ORMs, is fetching multiple columns, only to use two or three of them. Or discovering that MySQL often forces a disk sort when fetching rows with text or blob fields, even if you're not sorting on those fields. Or constantly updating only one column in a table when 20 other columns that never change, causing a lot of unexpected disk I\/O. For these and many other issues, most of which are easy to solve, devs who are unfamiliar with databases often reach for NoSQL solutions without understanding why they had a problem in the first place. Quite often they've traded a set of known problems for a set of unknown problems. This is not a good business decision.  More to the point: before relational databases, all databases were NoSQL. That's why we have relational databases.  Data is King  So why is this a big deal? Well, the term \"information economy\" isn't bandied about for nothing. Consider that most applications have four phases, though their structure might obscure this.   Initialization  Input  Calculation  Output   Virtually every application has those phases, but what do they mean? Simply put:   Initialization of data  Input of data  Calculation of data  Output of data   Your application is all about data. For many people in management, they see the app or the web page, but they don't see the database hidden underneath. They understand that without their data the company is probably in serious trouble, but they don't give much thought to it. In fact, while total data loss will probably bankrupt your company, few give much thought to the the standard data quality issues that are a constant financial drain because it's assumed to be part of the cost of doing business. And the developers who don't understand how to use databases often tell management \"that's just the way this technology works.\"  If there is one thing to take away from this article, it's to know that a properly designed database will still allow you to add incorrect data such as a misspelled first name, but it makes it hard to add invalid data, such as an order which references a non-existant customer, or a negative price for a product. Data maintenance costs drop significantly when you have a great database.  Even the barest minimum of work understanding how to design databases can tremendously improve quality. One of my most popular talks—rated 5 out of 5 stars at OSCON—is \"How to Fake a Database Design.\"     You don't need to be an expert in databases to do a much better job of using them. First, learn the basics of database design. Understand that foreign key constraints, check constraints, and other tools are part of that design, and not just \"nice to have\" features. Once you have your data quality ensured, then you can worry about performance. As the old adage goes \"first make it work, then make it fast.\"  NoSQL, sharding, and other techniques to get performance wins can wind up costing you a lot of money in the long run. Until you really understand why your current database isn't fast enough, don't reach for the shiny tools. You'll thank me later.  As an aside: I use and recommend NoSQL solutions from time to time (a shout-out to Redis: I love Redis). I am not \"anti-NoSQL.\" I simply find that by properly designing databases and understanding the underlying causes of database issues, NoSQL solutions often offer us little value.  ","title":"How the Database Can Hurt Your Startup","url":"\/articles\/how-databases-can-hurt-your-startup.html"},{"body":" Update: Many peoplea argue that messaging is flawed because your software breaks if you don't get a response. I wrote a new article, begging to differ .  I'm not going to link to the thread which set me off on this topic because there's no need to embarrass the people who say things like ...  Programming existed. Objects existed. Then [Alan] Kay contributed \"-oriented\" to the term \"object-oriented programming\" and the world went wild, but why?  Or another person, objecting to Kay's criticism of \"modern\" OO programming:  \"It's different than what I already know so it's wrong.\" This kind of thinking is why I quit CS major, chose to not pursue a career in programming, and haven't looked back.  This was largely in response to Alan Kay's famous 1997 OOPSLA keynote which is worth watching in its entirety.     There's much more in that thread, and while some understood what Kay was trying to say, many others seemed unaware of Kay's background and acted as if he was some grumpy old dude who just graduated from a Rails Bootcamp.  This is Dr. Alan Kay we're talking about! He doesn't have random opinions about \"objects\", he invented the word back in the 60s  . He saw what was happening in the programming world and was helping to craft many of these ideas, so he created a word to make these ideas easier to talk about.  And he's still bitter about choosing that word. It made people focus on the implementation rather than the behavior and it's all been downhill from there. Or as Dr. Kay put it \"I invented the term Object-Oriented, and I can tell you I did not have C++ in mind.\"  Today, most developers think OOP is about classes and inheritance. Some of the bright ones pipe up about encapsulation and polymorphism, but that's about it. class Dog isa Mammal {...} has won.  But for Dr. Kay, he states that OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things.  Why? Well, part of his background was cell biology and when he did the math on their computational power, he realized that while software routinely has trouble scaling, cells can easily coordinate and scale by a factor of over a trillion, creating some of the most fantastically complex things in existence, capable of correcting their own errors. By comparison, the most sophisticated computer software programs are slow, tiny, bugfests. Kay's conception of OOP starts with a single question: how can we get our software to match this scalability?  Isolation  First, let's discuss isolation. This is a shorter term than \"local retention and protection and hiding of state-process\".  The interior of a cell is messy and confusing, but the cell membrance wraps this up in a tidy package, hiding the internal details. It's estimated that around 50 to 70 billion cells die in your body every day. But you don't. Could your software keep running if you had millions of exceptions being thrown every minute? I doubt it.  You not dying when your cells die isn't encapsulation; it's isolation. Consider the following (awful) example:  class MyExample: def reciprocal(self, num): return 1.0\/num example = MyExample() print example.reciprocal(4); print example.reciprocal(0);  In the above code, we've encapsulated the reciprocal equation in the class, but then ...  0.25 Traceback (most recent call last): File \"class.py\", line 7, in &lt;module&gt; print example.reciprocal(0); File \"class.py\", line 3, in reciprocal return 1.0\/num ZeroDivisionError: float division by zero  The object dies, as does the code which contained it. This is the antithesis of what Dr. Kay is trying to get us to understand.  If you think of Web browsers and servers as objects, however, we see something closer to his vision.   If your browser crashed every time a web server crashed or was otherwise unavailable, Microsoft IIS would never have reached 2.0.  Now that we sort of understand a core idea of Kay's, Lets take it further. Kay points out that from the early days of Arpanet in the 60s, to the time of his OOPSLA keynote in 1997, Arpanet had grown roughly 100 million times the size of what it was.   And it didn't have to be repeatedly taken down for maintenance every time we wanted to extend it. The internet, today, is sometimes cited by Kay as the only working example of his OO model.    Extreme Late Binding  Another key point is his arguing for extreme late-binding. What does that mean? Well, consider this code:  my $order = OrderFactory->fetch(%order_args); my $invoice = $order->invoice;  If you have multiple \"order\" classes, you may not know which class you are dealing with, so you can't know, at compile time which invoice method you're calling. OOP languages generally don't select (bind) the method ( invoice ) for the invocant ( $order ) until run time. Otherwise, polymorphism can't work.  But what's extreme late binding? Does the invoice method exist? In a language like Java, that code won't even compile if the method doesn't exist. It might even be dead code that is never called, but you can't compile it if that method isn't there. That's because Java at least checks to ensure that the method exists and can be called.  For many dynamic languages, such as Perl (I &#9829; Perl), there's no compilation problem at all because we don't bind the method to the invocant until that code is executed, but you might get a panicked 2AM call that your batch process has failed ... because you might have encapsulation, but not isolation. Oops.   . This is \"extreme\" late binding with virtually no checks (other than syntax) performed until runtime.  Binding can also refer to binding a variable type to data.   . While this is a grotesque oversimplification of a complex argument, one way of looking at the difference between static and dynamic languages is that static languages such as Java often bind the data type to the variable (in other words, the named container of the data), meaning that you can never assign a different type to that variable, while dynamic languages such as Perl bind the data type to the data itself, meaning that the data knows about its type, but the variable often does not.   The latter is late binding because you often cannot infer, from static analysis, the data types at compile time.    Extreme late-binding is important because Kay argues that it permits you to not commit too early to the \"one true way\" of solving an issue (and thus makes it easier to change those decisions), but can also allow you to build systems that you can change while they are still running ! When was the last time you changed the behavior of a program and didn't have to stop it and restart it, often waiting hours for a complex recompilation? If you've worked with Smalltalk or Erlang you've probably had this pleasure, but most of the rest of us are still struggling with the hell of Makefiles and scheduled downtimes.  Messaging  We can kind of see the isolation and late-binding, but it's messaging where Kay's message seems to fall down for most, even though he's quite insistent that it's the most important thing there is in OOP. In essence, objects should be able to announce that they did things and other objects can ignore them or say \"hey, that's cool. Here's what I did in response.\"  But that doesn't really get to the core concept of messaging and frankly, this is the one area where I think Kay has been a touch vague, largely because many developers think that ideas are nice, but they want to see an implementation or proof of concept. In Kay's world. every object is almost like an entire computer, not just a unique area of responsibility, and each object can receive messages and figure out whether or not it knows how to deal with them.  In other words, you don't execute code by calling it by name: you send some data (a message) to an object and it figures out which code, if any, to execute in response. In fact, this can improve your isolation because the receiver is free to ignore any messages it doesn't understand. It's a paradigm most are not familiar with, but it's powerful.  There's a lot more to be said about this, but I'll stop and instead direct you to this article, written by Alan Kay, about The Early History of Smalltalk  . I actually found myself giddy reading that, getting a glimpse into the Cambrian explosion  of computing ideas which happened during the 60s and 70s. Sadly, computing, as with evolution, left that explosive era with powerful tools, but fewer ideas. Functional programming is slowly gaining more converts, logic programming is largely ignored   , but most programming today is procedural or class-based OOP. But Kay's ideas of what he intended for OOP are still with us. You can check out Erlang  and discover a marvelous programming language where code is isolated, late-binding, and passes messages around. And what do you get for that?   Extremely fault-tolerant code  High availability  Code whose behavior can be changed while the software is still running   That last point is a killer feature for many companies. The Erlang syntax and behavior is strange for many developers (it looks a lot like Prolog with the solver removed), if you have millions of customers counting on your service to always be there, Erlang is a powerful option.  OOP Today  Kay argues (correctly, IMHO), that the computer revolution hasn't happened yet because while our bodies are massively scalable meat computers, our silicon computers generally don't scale in the slightest. This isn't just because silicon is slow; it's because of things like print customers.fetch(customer_id).last_name not actually having a last_name method, throwing an exception (assuming it compiled in the first place) and programmers getting frantic late-night calls to bring the batch system back up. The only real upside is that it offers job security.  Sadly, this is where most of us are today. And would-be programmers get silly lectures telling them that class Motorcycle extends Vehicle , the instructor ( isa Person ) gets flustered explaining for the n th time the difference between a class and an instance, while a student ( isa Person ) in the back is noticing that a bicycle looks an awful lot like a motorcycle and the code she's looking at doesn't seem to account for this.  So class-based OO has won,   even with its limitations, arguments over inheritance versus composition, static versus dynamic typing,   and programmers arguing whether \"objects\" are data with behaviors attached or vice versa. So this sometimes puts me in an awkward situation when someone asks me what objects are since the myriad disagreements are strong, and valid.  I sort of punt on this, taking a \"meta view\" of what we're trying to accomplish when we write software. So I'll finish up by offering my view on objects, with an extended quote from my book Beginning Perl  .   ÆVAR THE PERSONAL SHOPPER  Many books have been written about OOP and even among experts, there is often disagreement about what OOP is. Many programmers have tried to explain OOP and leave the programmer confused. A case in point is the classic “An object is a data structure with behaviors attached to it.” Although that’s correct, that’s also an awful description and tells you almost nothing you need to know, so instead of giving you a textbook definition, we’re going to tell you a story.  You’re an awfully busy person and have little free time but plenty of disposable income, so you’ve decided to hire a personal shopper. His name is Ævar (any resemblance to reviewers of this book, living or dead, is purely coincidental) and he’s friendly, flamboyant, and most of all, cheap.  Because Ævar is new to both your city and the job, you have to tell him carefully how much money he can spend, exactly what quality of products you want, and where to buy them. You may even have to tell him which route to drive to pick up the goods and how to invoice you.  That, in essence, is procedural code and that’s what you’ve been doing up to now. You’ve been carefully telling the computer every step of the way what to do.  After a few months of explaining every little detail, Ævar gets upset and says, “þegiðu maður, ég veit alveg hvað ég er að gera” (Icelandic for “Shut up dude; I know what I’m doing”). And he does. He knows what you like and where to get it. He’s become an expert. In OO terms, you might now be doing this:  my $aevar = Shopper::Personal->new({ name => 'Ævar', budget => 100 }); $aevar->buy(@list_of_things_to_buy); my $invoice = $aevar->get_invoice;  You’re no longer telling Ævar every little step he needs to take to get your shopping done. He’s an expert, and he has all the knowledge needed to do your shopping for you and present you with the bill.   And that's it. Objects are simply experts. You tell them what you need and they get it done. Forget all of the handwaving about blueprints or \"data with behaviors.\" Those are implementation details. And once you start thinking about objects as simply experts about a particular problem domain, OOP becomes much easier.  ","title":"Alan Kay and OO Programming","url":"\/articles\/alan-kay-and-oo-programming.html"},{"body":" I may not win many friends by commenting on the emperor's nudity, but let's charge ahead anyway. I am not a clever man.  Possibly you've not heard of Douglas Hubbard . If you're interested at all in estimating the value of IT projects, his works should be required reading. In fact, his books are required reading for the Society of Actuaries if you want to pass their exams and become an actuary. Douglas Hubbard is a clever man.  But we'll come back to him in a moment.  If there's one key thing I've learned in consulting, it's that potential clients who start discussions by talking about building value for their customers have usually been better clients than those who start conversations with \"what's your rate?\" Why? Because the \"what's your rate\" clients are focused on controlling costs, not building value. I've found that their approach problems is different and those who are paying more attention to minimizing risk than maximizing value simply have different ideas about how to approach a project.  And you know what? I get it. If you want to go out to dinner and you only have £15 in your pocket, you're probably not going to a fancy restaurant. You just might grab a pint at a pub and hit a kebab shop after. Budget constraints are real.  But curiously, while development costs on a project are one of the first things we look at, they're also one of the worst metrics you can use to judge the value of a project. It's part of a problem that Hubbard refers to as the IT Measurement Inversion . Specifically:  The variables having the greatest impact on project success are the variables we measure the least.  So what sort of variables are there? There are the ongoing development and maintenance costs when the project is delivered. There are marketing costs. There are costs to have your people learn the new system. And so on. The costs are myriad and complex, but if we're going to judge the value of a new system, which variables matter the most? Those are the ones we should be focusing on.     Hubbard's work will guide you here. In particular, you might want to consider his book How to Measure Anything (that's not an affiliate link, by the way), where he teaches you how to break problems down to things that can be measured and how to measure them . You'll learn quite a bit about statistical analysis and running Monte Carlo simulations . In fact, one of the things you'll learn is that of the various techniques which have been developed to estimate project success, Monte Carlo simulations win hands down.  In fact, Monte Carlo simulations are so powerful that Hubbard talks about research at NASA showing that, for over 100 projects, the accountants outperformed the subject matter experts (scientists and engineers) on project estimates because the accountants used Monte Carlo simulations rather than other methodologies.  But for the purposes of our article, the key takeaway is that Hubbard has done a lot of this work for you and has nailed down what your real risks are. And it's almost intuitive once you think of it. Let's start with the most important.  The greatest risk to project success is whether or not it will be canceled.  That seems so brain-dead obvious that it's almost insulting to say, but when was the last time you quantified that risk? When was the last time you looked at the potential value of a large project and said \"there's a 25% that this project will be canceled\" and then factored that into your risk estimates? It's the single most important variable, but most people ignore it! Can you imagine running a company and discovering your managers are repeatedly ignoring the most valuable information at their disposal?  The next most important variable is system utilization. This is actually complex and includes how fast your product will roll out, whether people will use it, their rate of adoption, and so on. Again, this seems so obvious given 20\/20 hindsight. Sure, your project took six months to develop instead of three, but customers will be using it for many years. When it's put that way, it's obvious that estimating system utilization is more valuable than estimating development costs. But people still ignore it.  So let's look at development costs.  Of course you have to factor in development costs; I'd be insane to suggest otherwise. You have a budget and when you're looking at backing your project with Oracle or PostgreSQL, most of the time you find that Oracle isn't a terribly cost-effective solution. So yes, costs are important, but consider this: if the first version of Amazon's code cost twice as much to develop, Jeff Bezos would probably still be insanely rich. Uber would still be dominating their space. Facebook would still be selling your pesonal data to dodgy companies. I can confidently say that because these companies are providing real value to people (you might disagree about Facebook, but I can offer you 2.32 billion rebuttals ). The fact that these projects weren't canceled and are being used far, far outweighs the initial project costs. Or to restate all of this:  Initial development costs are incurred once. Your revenue stream is continuous. That's why development costs are less important.  \"But Curtis, I don't know how to do Monte Carlo simulations or estimate projected system utilization!\"  Well ... that's actually a fair point. These articles are intended, if nothing else, to be practical.  Since the two most importance variables in project success are the odds of it being canceled and system utilization, we need to reduce the risk of each of those. How do we do that?  Simply put, you go lean\/agile. Figure out the smallest possible work you can do which would add value and build that. Instead of a year-long project, you have a crazy one month race to get a simple prototype in front of your customers. If it turns out your customers have zero interest in your \"Recycled Food\" project, it's better to find out after a month instead of a year.  But let's say there's some interest. What now? Well, you've greatly reduced the chance of project cancelation because it looks viable, so you need to focus on system utilization. Amongst the key factors, getting customers to actually use the damned thing is incredibly important. You will be amazed at the ideas they come up with and the feedback they give you. As you push forward, you address customer concerns and have a lower risk of building features on top of things they say aren't working. You're constantly pushing new versions of the product in front of your customers and iterating based on their feedback. Every step of the way, you're improving your project based on customer behavior, not your hunches or market research.  This, incidentally, is why Lean Startups are such a powerful movement in business today. They assume they're not canceling their entire business, so they need to maximize system utilization. That means rolling the product out quickly and immediately responding to customer feedback. By doing this, assuming your product isn't canceled, you've now reduced the single greatest risk your project faces.  If you have any thoughts about this, please leave a comment below.  ","title":"Estimating Development Costs is Almost Useless","url":"\/articles\/estimating-development-costs-is-almost-useless.html"},{"body":"    Corinna in the Perl Core!  The Teams  Other Languages  What We’re Losing  You Can’t Violate Encapsulation  No Multiple Inheritance  No Inheriting from Blessed Classes  What We’re Gaining  The Principle of Parsimony  Conclusion     Corinna in the Perl Core!  It’s been a years-long, painful process, but with the release of Perl v.5.38,0  , the first bits of Corinna have been added to the Perl core. For those who have not been following along, Corinna is a project to add a new object system to the Perl core. Note that it’s not taking anything away from Perl; it’s adding a core object system for better memory consumption, performance, and elegance.  If you don’t have v5.38.0, you can still install Feature::Compat::Class  on Perl v5.14.0 and above and get the same features.  The MVP  for the Perl core is in seven stages, three of which are done and one of which (convenience attributes) is started (there is a :param attribute for field ).   Classes (done)  Inheritance (done)  Roles  Convenience attributes  Field initializer blocks (done)  MOP  Method modifiers   We need at least six of those to be implemented to really have something decent out of the box, but the MOP, while important, is not something that most OOP developers reach for on a regular basis.  You can read what’s currently implemented at perldoc perlclass  , but here’s a 2D point class using the existing features. We’ll make the default of the point be the graph origin.  class Point::2D { field $x :param = 0; field $y :param = 0; field $orig_x = $x; field $orig_y = $y; method x () { $x } method y () { $y } method reset () { $x = $orig_x; $y = $orig_y; } method move ($dx, $dy) { $x += $dx; $y += $dy; } method description () { return &quot;($x, $y)&quot;; } } my $point = Point::2D-&gt;new( x =&gt; 10, y =&gt; 20 ); $point-&gt;move( 5, 5 ); say $point-&gt;description; # (15, 25)  When the full Corinna is implemented, the method x and method y will be replaced with:  field $x :param :reader = 0; field $y :param :reader = 0;  The Teams  Here’s a partial list of the main contributors and\/or influencers to Corinna:   Damian Conway  Dan Book (Grinnz)  Graham Knop (haarg)  Harald Jörg  Matt Trout  Paul Evans  Sawyer X  Stevan Little   Note that Corinna was largely inspired by the work of Stevan Little. Damian Conway was also a huge contributor, but he was mostly behind the scenes sharing email with me and posting to blogs.perl.org  ;  Sawyer X’s main contribution was being the pumpking who encouraged me to stop trying to implement Corinna, but instead focusing on building something great without worrying about the current limitations of Perl.  Paul Evans is doing the actual implementation work and he’s doing a fantastic job.  The others (not to slight them, I can’t remember all the details of a multi-year project) were also huge contributors. They helped suss out design flaws, called me on my bullshit, and helped me understand the limitations of what we were trying to accomplish.  It’s also worth noting that due to this team effort, Corinna is not what I wanted. I wanted more. Much more. But it wasn’t just about trimming Corinna down to what P5P would accept, it was also vetoing many of my ideas about what I thought should be in Corinna. I was wrong about a lot of things and I’m sure I’m right about about some others, but the team effort is what made Corinna what it is today and I’m immensely grateful for that.  Other Languages  In the process of developing the Corinna MVP, many other languages were looked at. I was particularly curious about Beta  , but there was nothing that fit. I also looked at Smalltalk, Java, Raku, Python, Eiffel, and Ruby (and probably others I’m forgetting right now). There are many interesting ideas, but given that we were trying to get something that fit Perl, the biggest influences were Raku, Moo\/se, and yes, Java. Primarily, I was interested in protected methods and inner classes, but the former seemed problematic and the latter wasn’t something appropriate for the MVP. However, reading about OOP design (so many examples are in Java), helped quite a bit with Corinna.  What We’re Losing  There are, of course, trade-offs in something this huge.  You Can’t Violate Encapsulation  The biggest concern amongst developers is that we can no longer violate encapsulation (well, you can via the MOP, but it will be more work, as it should be. Emergency patches sometimes require violating encapsulation).  However, I don’t know many developers who can define encapsulation. People keep saying it’s about hiding state, but that’s not true. It’s about hiding state and process because the two are intimately linked in classes. If you can change the state without going through the interface, you have no guarantees the process is correct. That’s a bad thing.  Fortunately, this has been well-understood for decades, so there’s a wealth of information about proper OOP design on how handle this. I expect there will be changes to Corinna to help with this, but we don’t know what they are yet.  No Multiple Inheritance  To my surprise, no one on the design team strongly argued against this change. There were discussions, but nothing too serious. There are many OOP languages which do not allow multiple inheritance and it’s not a big deal. In fact, there are some OOP language which don’t allow inheritance at all.  Most languages which forbid multiple inheritance offer some sort of alternative. In our case, it’s roles.  No Inheriting from Blessed Classes  Your new Corinna class cannot inherit from anything not declared as a class . This was a slightly contentious issue, but there are simply too many differences. Corinna doesn’t allow you to “reach inside” and manipulate the data. Corinna is single-inheritance, so it’s unclear how that would have worked with classes allowing multiple inheritance. And ultimately, we’d like Corinna to have the option of having a different UNIVERSAL:: base class, offering different capabilities.  This, of course, means that developers will have to understand “composition over inheritance.” This is a good thing.  What We’re Gaining  Object-oriented programming will be “first class” in Perl, not just a hacked on solution. Methods and subroutines will no longer be the same thing. We’ll have private methods. We already have a much more concise, and clear way of writing classes. Companies using the Object::Pad  prototype have reported a 10% reduction in CPU usage and that’s without optimization. We have have potential for a much more efficient implementation of objects than we’ve had before.  The Principle of Parsimony  Given all of the above, I have confession to make about the new class syntax for Perl: we made a mistake. It’s a whopping huge mistake.  The problem is that we don’t know what that mistake is. There’s probably more than one of them.   In the README of the Corinna repository is a section entitled The Principle of Parsimony  , it states:   Many things in the proposal are deliberately restrictive, such as Corinna only allowing single inheritance. This is to allow Corinna to be cautious in not promising too much. If we later find this too restrictive, we can allow multiple inheritance. However, if we start with multiple inheritance and discover we don’t need or want multiple inheritance, we would break existing code by taking it away.  Any proposals to change the RFC must consider the principle of parsimony.   The design team really should have highlighted this more because it’s the single most important thing (outside of the new class syntax itself).   Most people seem happy with the changes. They’re new. They’re exciting. They’re a step towards a new future for the language.  Some people are unhappy. Some are unhappy for reasons that are, well, silly. Case in point: I’ve had a developer angrily tell me they’re nearing retirement and they don’t want to keep up with a bunch of changes in the Perl language (more than one, but the other wasn’t quite so strident in this position).  Some are unhappy because Corinna didn’t go far enough.  Some are unhappy because Corinna has gone too far.  Others are unhappy because « insert reason here » .  I get that. I really do.  So why doesn’t Corinna have multiple inheritance? Because if it did and it’s a mistake, we can’t easily walk it back without breaking existing code.  So why doesn’t Corinna allow inheriting from non-Corinna objects? Because if it did and it’s a mistake, we can’t easily walk it back without breaking existing code.  So why doesn’t Corinna have native support for field (attribute) builders like Moo\/se? Because if it did and it’s a mistake, we can’t easily walk it back without breaking existing code.  The counter-argument is that the new class syntax is experimental and we should have that freedom. My counter-counter-argument is that I know of large code-bases which haven’t upgraded past 5.26 because 5.28 swapped the order of attributes and signatures bodies  .  Companies can and do rely on experimental features.  The design team agreed (generally) to follow the Principle of Parsimony to ensure we didn’t paint ourselves into corners more than we already had. We expect that in the real-world, people are going to make compelling arguments for changes based on limitations they discover. We know mistakes are going to be found. We’d like to see if those are mistakes in our design, or mistakes in how people (ab)use OOP code in Perl.  I might add that the Principle of Parsimony hasn’t been an absolute, even in the design stage. For example, originally Corinna required a postfix block:  class Foo { ... }  The extremely tight scoping of the postfix block would make it much harder for Corinna keywords to bleed into an unwanted scope. I argued for that, but I was overruled. The postfix block is not required.  So please keep this in mind about future discussions of the new class syntax.  We made mistakes. We don’t know what they are. We worked very hard to try to ensure that fixing those mistakes means extending Corinna, not breaking it.  Conclusion  Corinna, for now, is experimental and incomplete. The biggest obstacle for its adoption is likely to be the fact that developers are going to have to learn new ways of programming. The most annoying might be that field variables aren’t available in subclasses (encapsulation!), so you’ll find it convenient to make a public accessor to fetch this data. There’s a similar issue with roles. The Java notion of protected methods is not available.  There are ways we could potentially address these issues, but we need to see what people are doing with Corinna and what their real pain points are.  Fortunately, the Object::Pad project shows that the new approach plays quite well with existing code, so we are pretty sure we have the basics right, but we don’t know what we don’t know.  Interesting times are ahead.  ","title":"Corinna in the Perl Core","url":"\/articles\/corinna-in-the-perl-core.html"},{"body":" A fellow agile trainer recently told me that many of his customers are fleeing scrum to get away from the zealots. I sighed and nodded in agreement—all too familiar with those agile enthusiasts who repeat the \"agile isn't a silver bullet\" mantra—and then proceed to recommend it for everything.  Most would agree that not everything must be agile, but when you ask for examples of what the agile methodology isn't good for, you get blank stares. To figure out when you should and shouldn't use agile, you need to understand how products evolve in the market, determine where your product fits on that continuum, and reexamine agile in that light.  The four phases of product evolution  Products tend to evolve through four maturity phases. Each has unique characteristics and should be treated differently. They include:   Novelty: This is the \"gee whiz\" phase, where a new and exciting idea emerges and people begin to wonder what can be done with it. For example, when online shopping first emerged, people wondered if you could sell clothes, food, or cars online. Would people trust you with their credit card information? Was there really a market for people buying goods from faceless, anonymous entities online? We know the answers now, but we didn't then.  Stability: This is the phase when people start to understand what they can do with a given product. The focus shifts from what can be done to how to do it. Remember all those \"how to build a shopping cart\" books and articles in the late '90s and early '00s? This was online shopping transitioning from novelty to stability.  Commodity: This phase kicks in when the market has stabilized enough that for most companies, building their own version of a product doesn't make economic sense. Instead, there are enough competing products out there that acquiring the product off-the-shelf is simpler and cheaper.  Utility: This is where you no longer purchase a product, but you consume it as needed, just as you would with electricity. Shopify, Etsy, and many other e-commerce utility solutions are readily available.   Of these phases, the novelty and utility phases are the most interesting. What makes the novelty phase interesting is obvious—startups make fortunes in this area. However, the utility phase is also fascinating if you're the one providing the utility. Products in the utility phase tend to be high-volume, low-profit systems. Entire industries are built on them. Electricity started out as a novelty and today is a well-established utility with industries such as television, radio, and computing built on top of it. Cars are going through this phase: with the rise of Uber, Lyft, and self-driving cars, the market is facing the prospect of cars becoming utilities. Someone is going to make billions when they figure out how to exploit the new markets built on top of this change.  One characteristic of the novelty phase is the rapid change and uncertainty. This doesn't exist for the utility phase—customers will be reluctant to consume your utility if rapid change and uncertainty is another price they have to pay.  Agile's job: To reduce the cost of change and uncertainty  Once you understand where a product is in its evolution, the appropriate use of agile should become clear. Agile is nothing more than a method of reducing the cost of change and uncertainty.  This is reinforced by the fourth point in the Agile Manifesto: \"responding to change over following a plan.\" In fact, while many have read the Agile Manifesto, few have read the twelve Agile Principles associated with it. Many of the principles reinforce the point that agile is about embracing change and lowering cost. If you think agile is all about standups, short iterations, retrospectives, story cards, and so on, then you've missed the point. You're not alone: many practitioners have been misguided because agile is often taught as a set of behaviors, with little reference to the goal beyond vague mentions of side-effects like improved time to market, customer satisfaction, and lower startup costs. It's like teaching people the moves of chess and not teaching them about checkmate. In this light, scrum, Extreme Programming (XP), Kanban, and other methodologies are tools you use after you've already decided on an agile approach.  So when should you choose agile? When you understand both agile and product evolution, the answer is simple: the newer the product and its market, the better fit with agile.  Agile works best in environments with the expectation of change, when you need to optimize for uncertainty. Products in the novelty phase definitely benefit. However, what if you pretty much do know everything up front? What if there is little change and uncertainty in your environment? One agile mantra is to \"fail fast.\" Do you really want to fail fast if you're building a nuclear reactor? Of course not. Nuclear reactors are one attempt to implement the utility of electricity. Agile isn't scrum, XP, or Kanban. Agile is merely a way of reducing the cost of uncertainty and change. If your project is close to the novelty phase of its market evolution, you'll have a lot of uncertainty and change. In this situation, agile (reducing the cost of change) is a great choice. But if your product falls into the stability and commodity phases, consider switching to a lean methodology (minimum viable feature set with rapid development, reducing the cost of waste). Finally, if your products are commodities that are perhaps evolving into utilities, consider techniques such as Six Sigma (reducing the cost of inconsistency). Why? Because they focus on reliability and reproducibility. Optimizing to lower the cost of change and uncertainty doesn't make sense when you already have little change or uncertainty. Not for everyone  Many agile practitioners object at this point. They insist that you can simultaneously optimize for change, uncertainty, reliability, and reproducibility. For those people, here's a thought experiment. Imagine you've decided to create a tool to allow job applicants and companies to rate recruiters. Using crowdsourcing, you can start to identify the best recruiters for your needs. Put five excellent, independent agile teams on this project and you'll have five different products at the end. Reproducibility simply isn't something agile was designed for.  Don't get me wrong: I'm a huge fan of agile. I've happily embraced it for years, and I enjoy working in agile environments. But it's time we end the tyranny of agile and recognize that it's merely another tool in the toolbox. Agile is a great hammer, but not everything is a nail.  Note: I originally published this article on TechBecon .  ","title":"When to Choose Agile","url":"\/articles\/when-to-choose-agile.html"},{"body":" My discussion of Alan Kay's ideas behind OO programming made the rounds, getting about 13,000 page views (as of this writing) from various sources, including Hacker News  and Reddit  . There was quite a bit of discussion, most of which I'll cheerfully let people decide for themselves, but there's one perennial question, as exemplified by this response  , which I really need to address.  So, what exactly should happen when you request last_name on an object that doesn't respond to that message? ... It tells me you haven't been the one receiving those 4AM phone calls, angry dictates from C-levels, and responsible for hundreds of thousands of dollars (or even millions) of lost revenue per minute of down-time.  There was a lot of vitriol in the comment and I didn't want to respond to that, but in this case, that vitriol actually strikes to the heart of this issue. Specifically, I have been on pager duty for several companies, getting those frantic early morning phone calls and that's helped to guide my thoughts on what robust computing should be, and how you might handle objects that don't respond to messages.    When you don't handle messages gracefully. Source   Back in the 90s, when I was a COBOL programmer for a huge insurance company, I was responsible for writing software that would integrate third-party data into our accounting system. Whenever I would write new software, I would send that to our \"operators\", along with a series of steps on how to restart the software if any step in our JCL  failed. I really didn't want to get a late-night call, so I tried very hard to ensure those instructions were clear, and even took the naughty step of allocating more disk space than my software needed to guarantee that this would never be a problem with my software. I knew the main accounting software ran out of disk space every couple of weeks, requiring costly work on the part of the developers and operators to coordinate on resurrecting a batch job.    Obviously, that was much earlier in my programming career. I didn't know anything about OOP, had never heard of Alan Kay, and couldn't possibly understand the core issues he was discussing, but I had done one thing right: I wrote down very, very complete documentation that intelligent people could use to restart my software if it failed. Keep that in mind: I gave a lot of thought about how to respond if my software didn't perform as expected.  Fast forward a couple of decades to when I was hired to fix the ETL system of a company working to reduce the costs of Phase III clinical trials.  .   It was a large project, with many parts, but the ETL system had been written a couple of years earlier by a doctor! I was rather surprised. It was clear that he wasn't an experienced developer, but his work showed a lot of brilliance because he deeply understood the problem space.  The ETL system was a bottleneck for this startup and honestly, no one wanted to touch it. A new dataset would arrive from a pharmaceutical company and it would take two days of manual effort from a programmer to get the dataset into a shape where it could be run through the \"ETL\" system. I say \"ETL\" in scare quotes because obviously you don't want to take a highly-paid software developer off of their work for several days of manual data cleanup just so you can process an Excel document. But this work was critical to the startup and while they dreaded getting new data, they had to handle it. So they offered me a contract and the other developers wished me luck, happy to be washing their hands of the mess.  After a few months, I had rewritten the ETL system to be a fault-tolerant, rules-driven system that could handle the radically different data formats that pharmaceutical companies kept sending us.   More importatnly, instead of two days of hands-on work from one of our developers, it turned into two hours of automated work, leaving the developers free to do other tasks. So given that the data quality was rubbish, how did I get there?  I had to jump through a lot of strange hoops, but here's an interesting key: the pharmaceutical companies were astonished that we could get a year's worth of work done in a month. They simply were not able to change their processes and asking them to \"fix\" the data they sent us failed. So my job was to fix it for them. And that brings me back to the main issue with Alan Kay's idea about messaging. What do you do if you send a message and don't get a response? Here's how we programmers tend to think of that:  try { \/\/ This rubbish method names helps to illustrate that we \/\/ might have an unreliable source result = someObject.fetchDataFromServer(orderId); } catch(UnknownOrder uo) { System.out.println(\"Unknown order id: \" + uo); } catch(ExistentialCrisis ex) { System.out.println(\"Server is sitting in the corner, crying.\"); } catch(Exception e) { System.out.println(\"Yeah, this blew chunks.\"); e.printStackTrace(new java.io.PrintStream(outputStream)); }  This is the software equivalent of \"Not My Problem.\" After all, the fetchDataFromServer() method should return data! It's not my fault if the other team's server sucks!  But I had no way of telling a bunch of multi-billion dollar pharmaceutical companies to get their act together. Making the ETL system work very much was my problem. And by taking responsibility for that problem, I wrote a much more robust system. So what do you when fetchDataFromServer() fails? You do the same thing I did at the insurance company when I sent instructions to the operators explaining how to \"fix\" things when the software failed. Except that I'm sending that to my software, telling it how to fix itself. This requires teaching developers a different way of thinking about software, something that goes beyond try\/catch , \"not my problem.\"  So when fetchDataFromServer() fails, or when an object doesn't respond to the message you sent, what do you? There's a simple way to approach this: ask yourself what you would do in real life. What if you deleted your partner's phone number by accident and you need to call him. Do you just give up and say \"not my problem?\" Of course not. If he's ever sent you a text, you probably have his number on the text. Maybe he sent you an email containing his number. Maybe your friend has his number and you can ask them. Or maybe you can find a different way of contacting your partner. The choice is yours: give up, or take responsibility for finding a solution.  Hell, maybe we should call this RDD, \"responsibility-driven development,\" because what the programming world needs is new acronyms.  For the ETL system, I would have problems such as needing to know if \"Dr. Bob Smith at Mercy Clinic\" was the same person as \"Robert Smith (no title) at Mercy Hospital.\" Sometimes those different records would come in from different pharmaceutical companies, but sometimes they would come in from the same pharmaceutical company; I'd still have to figure out if they were the same person. This is the very definition of \"bad data\" but it was critical that the ETL system work and, as it turns out, it's a fairly well-understood problem and there are many approaches.  In this case, I would use various geolocation services to find addresses and lat\/long data for the locations (with fallbacks if a service was down). I would create known \"name variants\" (in English, \"Robert\" is often shortened to \"Bob\"), encode names with something like Soundex   , stripped stop words from names, check email addresses and phone numbers, along with their Levenshtein distance since they were a frequent source of typos, and so on. By the time I was done, I had a large list of things which were potentially identifying, and scoring to indicate which sources of \"truth\" were more valuable.  By calculating a score, if it exceeded a particular threshold, we assumed they were the same person. Below a threshold, we assumed they were different people. Between those thresholds, we flagged them for follow-up. And if the external geolocation services were down? Fine! I just didn't get a score for their data, but I still had plenty of other scores to help me calculate that threshold.   Surprisingly, we usually had no follow-up records, despite often processing tens of thousands of people per batch.  This is how you deal with unanswered messages. You think about your response instead of just letting your software crash. Is it OK if you don't have that info? Can you just leave a spreadsheet field blank? Are there other sources for that information? Can you try later to fetch that information, having designed a nice async system to make this easier? There are tons of ways of dealing with these issues, but they all involve more work, and different problems, of course, involve different solutions.  This isn't to say that there's always a neat answer, but curiously, as I find problem spaces increasing in complexity, I sometimes find that there are more solutions to bad data instead of just \"more bad data.\"  There is, however, a little bug in this ointment: clients often want software today, not tomorrow. As we're pushed harder to develop more software in less time, struggling under the tyranny of budgets , we often don't have the time to make our software this robust. Even if it would save the client a lot of money in the long run, they want to move quickly and let their later selves pay down the technical debt. So I'll be writing a more try\/catch blocks in the future, because sometimes it's really not my problem.  ","title":"Alan Kay and Missing Messages","url":"\/articles\/alan-kay-and-missing-messages-a-follow-up.html"},{"body":" Note: The Model, View, Controller pattern, or \"MVC\", actually comes in many flavors, but for the purposes of this article, I'm referring to MVC for the Web. Feel free to comment below, but keep in mind that we have a particular viewpoint here.  The Problem: getting MVC wrong is expensive and costs you a lot of money.  I often start working with a new client as a manager or developer and find a Web app with a common set of mistakes . On one hand, I would like to see software developers learn from past mistakes. On the other hand, this pays my bills. In particular, MVC&mdash; which has unfairly been getting a bad rap &mdash;is thoroughly misunderstood and much of this can be blamed on rubbish examples and tutorials.  Here's a perfect \"bad\" example from an MVC explanation using Python and Flask :  @app.route('\/') def main_page(): \"\"\"Searches the database for entries, then displays them.\"\"\" db = get_db() cur = db.execute('select * from entries order by id desc') entries = cur.fetchall() return render_template('index.html', entries=entries)  In this example, we have a page which finds \"entries\" and displays them on a Web page. It's entirely possible that you see no problem with this, but it perfectly demonstrates what's wrong with people's understanding of MVC: you've pushed knowledge of your database, your business logic, and your HTML into the controller.  That's what our examples and tutorials are teaching developers. Because of this, when I work with a new client using \"MVC\", what I really see is this:  def some_controller_method(): \"\"\"begin hundreds of lines of code\"\"\"  How do I hate thee? Let me count the ways. (with apologies to Elizabeth Barrett Browning ). The following comes from years of experience seeing this over and over again.   The database or ORM is exposed at the controller level (nooooo!!!!!)  You can't reuse that logic outside of the controller  You probably can't test that logic outside of a Web context  Tests are much slower because of the web context  Test coverage suffers because it's hard to test conditional logic in the controller  Many parts of that logic get duplicated in other controller methods  Different behaviors become tightly coupled, making it hard to develop  Performance suffers because you can't cleanly target a layer  Integrating new technologies is harder because the layers are tightly coupled  ... and so on   But what does this mean in practice? Let's consider a real example I encountered a few years ago.  One company I worked with had all of the above problems. In particular, they had a 3,000 line \"method\" for searching for products. It conflated many things, including searching for products and filtering the search results. Thus, even though I was tasked with working on it, every time I touched search, it would break filtering. Every time I touched filtering, it would break search. It was slow and tedious to get anything done. It took me some time to separate search and filtering and when I was done, I had a surprise visit from one of the developers of the mobile side. It seems they wanted a proper search engine for ages but never added it because they couldn't separate the searching (which they needed) from the filtering (which they didn't). They thanked me because, after my work, it took them 15 minutes to implement a search engine.  Not getting this right flushes money down the toilet . Often we hear that a new feature will take a lot of time to implement \"because the application wasn't designed for this use case\" when what is really meant is \"because we didn't understand separation of concerns and thus a one week feature will take months.\"  Note:  you can fix this mess, but it takes time and you have to understand what MVC should look like.  A Proper Controller Method  To contrast this, let's look at a the controller for an extremely complex bit of logic from the free MMORPG, Tau Station , using Perl and the Catalyst MVC framework :  sub station ( $self, $c, $station ) : Local Args(1) { $c-&gt;character-&gt;travel_to_station($station); }  Without going into detail about what travel_to_station does (trust me, it's complex), all we see is a request for a particularly complex action to be performed. In the controller we can see:   No knowledge of how the model is constructed  No knowledge of how the data will be used  Knowledge of the route to this controller method (the Local attribute)   Having the route information hard-coded is a bit unfortunate, but that's easy to work around. However, this controller method merely connects the model to the view, and that's all. That's what you're looking for in MVC. MVC, when appropriate, takes an application with a UI (user interface), and decouples the responsibilities into reasonable layers.  It's now easy to test the logic (you're testing the model, not the controller layer and all of its web context). It's easy to reuse this code. And, if the model is properly built, you have a great separation of concerns that makes it easy to extend, maintain, and repair the application.   Model  The application.  View  What the consumer (user) sees. Often HTML, JSON, or XML.  Controller  A thin layer connecting the View and the Model.   And that's it! There's nothing fancy, but it often requires explanation. The controller, as shown above, should be as thin as possible. The view should only have display logic and the model is ... what, exactly?  The Model  I'll write more about this another time, but the model is the part where everything breaks down. Almost every MVC tutorial that I see confuses the data model with the Model. This is what we get (pseudocode):  method neighbors(): model = this.get_orm(); neighbors = model.table('Characters') .search( location: this.character.location ); this.template(\"people\", people=neighbors);  Now that looks reasonable. In fact, it seems so sensible that I did something very similar in the Tau Station MMORPG a long time ago. But it was deeply flawed. It turns out there are many cases where you need to know the other characters in your vicinity and this logic should be encapsulated. But it's just one method call, so factoring it out is silly, right?  Wrong. As it turns out, you didn't want to show any characters who hadn't been active in the past day. Well, unless they were NPCs. Or the other characters have something else which makes them \"non-visible\". Any time we discover a new case of when characters are or are not visible, we can visit this logic in every controller method, or we can abstract it out into a single method:  So let's try it again:  method neighbors(): model = this.get_orm(); neighbors = model.table('Characters').visible_to(this.character); this.template(\"people\", people=neighbors);  Ah, that's much better. Except we have exposed our ORM (and thus, our database). We have many data sources which have data in Redis, or configuration files, or a custom cache, or from all sorts of locations that the controller should never, ever know about. Some of my clients with this anti-pattern have worked around this by trying to implant those other data sources directly in their ORM, which simply shuffles the coupling around. The controller should never care about the source of data. The controller should only be asking for the data. Instead:  method neighbors(): neighbors = this.model('Characters').visible_to(this.character); this.template(\"people\", people=neighbors);  Where do the \"visible\" characters come from? You don't care. By hiding all of these details from the controller, the developers are free to implement the fetching of this data any way they want, so long as they respect the contract originally specified when this method was created.  The model is your business logic. And I'll go a step further and say a good design choice is to make the model provide services the client needs, with a headless application underneath. By creating a this, you can layer a controller and HTML view on it. If you're careful, you can have your controller target multiple views and build a native mobile application on top of it, only requiring a new view. You could create a native client\/server desktop application, too. If the controllers are too inflexible, that's OK. They're tiny and easy to skip. Write thin controllers specifically for your mobile and desktop apps, secure in the knowledge that the application logic is all in your headless application.  There any many benefits to this architecture and I'll cover them in a later article. Further, the headless application needs its own architecture. When is the last time you saw an MVC explanation go into detail about the architecture of the model? That's a rare beast indeed and it's why so many companies have themselves mired in technical debt: they get a partial explanation of a complex concept and start building from there..  ","title":"Fixing MVC in Web Applications","url":"\/articles\/fixing-mvc-in-web-applications.html"},{"body":"    Introduction  Inheritance  Classes versus Prototypes  Dr. Alan Kay  Messaging  Isolation  Extreme Late Binding  Microservices  A Little Secret     Introduction  I’ve been meaning to write this article for a while, but never quite got around to it. Now I’m around it and I know that it will annoy a few people. In fact, any time I read new discussions about how Dr. Alan Kay, the inventor of the term “object-oriented programming”, intended something entirely different from what we have today, the online discussions often devolve to people acting like he’s some grumpy old dude who just graduated from a PHP bootcamp.   If you want some extended discussions I’ve written about Alan Kay and OO Programming and Alan Kay and Missing Messages . They’re not required for following along in this article.   In 1967, Simula 67 was released. This is often referred to as the first “true” object-oriented language because it included these four feature:   Classes  Polymorphism  Encapsulation  Inheritance   It offered many other features, such as subtyping, coroutines, and discrete-event simulation  , but those are just curiosities for the purpose of this article.  Inheritance  Of the above list, the first three are generally non-controversial, but inheritance has been a pain in the ass for over five decades now. The following illustrates the core problem with inheritance:  class Person :isa(Invoice) { ... }  What the hell does that even mean? I suspect (though I can’t prove), that the majority of issues we have with inheritance today stem from that core problem. A subclass is intended to be a more specific instance of a parent class, but that’s a semantic issue the programming language can’t determine. Instead, there’s a kilo of fat shoved in our skull which has to figure out what that means and we often get it wrong.  This is problematic enough that Alan Kay left inheritance out of his original descriptions of Object-Oriented Programming (OOP). It’s problematic enough that some OOP langauges (or OOP-like, depending on your definition) don’t allow inheritance at all. Some allow multiple inheritance. Others only allow single-inheritance but usually give you some tools to make life a touch easier than simply composition and delegation (interfaces, mixins, aspect-oriented programming, roles (Smalltalk-style traits), etc.).  Importantly, in class-based OOP languages, the classes define the behavior and the instances of those classes contain the data (hand-waving on this quite a bit, of course).  Classes versus Prototypes  But what about the other features? They seem good, right? Well, there are a number of protype based programming langauges  , such as Javascript, Lua, Self, Io, and so on. In these languages, an object can contain both data and behavior and you copy the object and add new data and behavior as needed. To be honest, though, when I squint, I see single inheritance in protype-based languages. I’ve never worked with one enough to be confident in saying that, though.  Dr. Alan Kay  But here’s someone who should be considered an expert on the term. Note that he didn’t invent OOP, merely the term. And he’s not happy with where it went.     This isn’t a classical “argument from authority” fallacy because I’m not going to argue whether or not he’s correct. Instead, I’m going to argue that microservices adhere much more closely to his idea of OOP than traditional class-based systems.  In particular, he identifies the following three key features as the core of his idea of OOP  :   Messaging  Isolation  Extreme late-binding   All three of those are unusual enough to the minds of many programmers that they’re worth digging into. I’ll be brief.  Messaging  In Kay’s view, you don’t call methods on an object, you send messages to it. The receiver is free to respond however it likes, including ignoring the message. For example, with a web server, you could send the message DELETE \/foo.html and it’s free to return a 400, a 404, another status code, or simply drop the request. If you think of the server as an object , you’ve sent it a message and it can do any damned thing it wants with it.  Isolation  Alan Kay actually said, “local retention and protection and hiding of state-process,” but we often use the term “isolation” instead.  First, let’s think about encapsulation. If, internally, an object has a secret key for encrypting\/decrypting something, you don’t want that exposed. That’s encapsulation . Many languages get this right, though I’ve found that many dynamic languages make this hard to enforce. Fixing this is one of the key drivers of the Corinna OOP project for Perl  .  But what happens if we do this (Corinna OOP syntax)?  class User { has $username :param :reader; has $password :param; } my $user = User-&gt;new( username =&gt; &quot;Ovid&quot;, password =&gt; 'hunter2', ); say $user-&gt;username; # &quot;Ovid&quot; say $user-&gt;password; # &quot;Method not found&quot; error  So I can instantiate a User object, send it to you, and you can see their username, but not their password. That’s encapsulation. But if you try to call $user-&gt;password , you also get an exception. That’s encapsulation, but not isolation. Alan Kay referred to “state-process”, not just state.  Again, think about a web server and a web client. If the server throws a 500 error, or just catastrophically crashes, your client doesn’t suddenly crash (unless it’s poorly written). Instead, it gives you useful information which you can then use to decide what action to take. In fact, Kay has described the Web as being the most successful example of OOP ever invented.  Extreme Late Binding  In many OO languages, we have early binding where the class is used to resolve, at compile time, the method that is to be called. You will get compile time errors if the method does not exist.  In late binding, we use the instance to determine resolve, at runtime, the method that is to be called. If you don’t know what kind of instance you have, you can get runtime errors if the method does not exist.  So what’s extreme late binding? The method might not even exist when it’s called, but it still resolves, or its behavior might change between calls because you’ve found a more efficient or accurate implementation! Smalltalk and Erlang are two languages known to easily support this. Surprisingly, Perl can as well, using the AUTOLOAD feature. Here’s a silly example using the Corinna syntax, with a few speculative (made-up) features to make the example clearer:  class SomeClass { has @objects; ADJUST { # populate @objects } method AUTOLOAD ($method, @args) { foreach my $object (@objects) { if ( $object-&gt;can($method, @args) ) { $MOP-&gt;class-&gt;add_method( $method, method { $object-&gt;$method(@args) } ); return $self-&gt;$method(@args); } } return; } } my $instance = SomeClass-&gt;new; # return $object-&gt;frobnicate(42) from the first composed object # which can handle the request $instance-&gt;frobnicate(42);  In the above example, SomeClass doesn’t implement any methods at all, but if you call one that doesn’t exist, it installs one that delegates to an object having that method. Later calls would call the installed method instead of AUTOLOAD .  We can again use the web server\/client analogy to explain this. A browser makes a request and the web server may not even be designed to handle the request but serves a result anyway. It might even later cache that request so it doesn’t have to recompute it. You don’t know. You don’t care. You only know you made a request and got a response. Hopefully it’s a useful response.  The intent behind extreme late-binding is to protect you from committing too early to an implementation that may not be what you need (or even needed), but allow you to respond to the method\/message if appropriate.  Microservices  There’s a lot of confusion about microservices and SOA (Service-Oriented Architecture) and I cover some of it in this article  .   Microservices are generally expected to have all the logic and data they need to perform a task. They’re not coordinating with a bunch of other services and thus are loosely coupled and fit well with agile development.  SOA services, however, need to coordinate with other services in the same way that your Order object needs a Collection of Item objects, along with a Customer object, and so on.   A microservice is self-contained and might duplicate code in a way that SOA does not, but it’s completely isolated from the rest of the code. It’s basically a process\/server running that has an API (messages), isolates everything , and can use extreme late-binding if it wishes to.  Thus, if you aspire to Dr. Kay’s model (and the success of the internet is a powerful testimonial), you may want to look at microservices. Some people look at microservices as being a silly fad, but they’re a fad that’s been around for quite some time and more and more companies are using them. In fact, our company, All Around the World  is routinely getting more requests from clients about implementing SOA\/microservices for their systems.  One interesting point about microservices (which SOA might struggle with) is that the extreme isolation of them combined with an OpenAPI interface means that you can separate off a layer of your system as a microservice, make sure it works, and then easily rewrite it in another programming language that might be more appropriate to the task at hand. That’s often extremely hard to do when you directly use\/import\/require\/include a module because it’s usually required to be in the same language.  A Little Secret  But if you agree with the idea that microservices might be the next step in OOP, along the lines of what Alan Kay proposed, perhaps we should keep this our little secret, shall we? OOP “purists” might be horrified at the description and OOP “haters” might be aghast at realizing they’ve been writing objects all this time. Admitting that microservices are objects might be a marketing disaster, even if it’s true.  ","title":"Are Microservices the Next Phase of Object-Oriented Programming?","url":"\/articles\/are-microservices-the-next-phase-of-object-oriented-programming.html"},{"body":"     We say what we do and do what we say.    Note: I’ve written about this before, but the writing is scattered over the web and in conference keynote presentations I’ve given. This is to bring it home to one site. This code written by All Around the World  is driving the narrative sci-fi MMORPG Tau Station  . And if you feel the need, feel free to email us to discuss how we can build a remote software team for you. This is the quality we have been delivering for years, both for Tau Station and for our clients and therefore the quality you can expect for your own project.  Just about anyone who’s taken a beginning database course learns about the critical importance of database transactions. For example, consider the following code:  sub transfer_money ($self, $from, $to, $amount) { if ( $from-&gt;balance &lt; $amount ) { Exception::InsufficientFunds-&gt;throw(...); } $from-&gt;dec_balance($amount); $to-&gt;inc_balance($amount); }  The code is a grotesque oversimplification in several ways, but let’s just look at one little line:  $from-&gt;dec_balance($amount);  What happens if that fails and throws an exception? If it fails before the money is withdrawn, that might not be disastrous. If it happens after the money is withdrawn, we withdraw the money, but don’t deposit it to the new account. Our customers will be disappointed, to put it mildly.  So we fix that with a database transaction. It might look like this:  sub transfer_money ($self, $from, $to, $amount) { if ( $from-&gt;balance &lt; $amount ) { Exception::InsufficientFunds-&gt;throw(...); } $self-&gt;db-&gt;do_transaction(sub { $from-&gt;dec_balance($amount); $to-&gt;inc_balance($amount); }); }  Now if dec_balance fails, we’re guaranteed that whatever money was withdrawn will still be in the original account.  But it still has a massive bug. What if money was withdrawn after we checked the balance, but before we withdrew it? Oops. It’s called a “race condition”, because the behavior of your code depends on which code finishes first.  In this case, the behavior depends on whether the system allows negative balances, whether the account allows negative balances, and other factors. We don’t want to risk this bug, so let’s expand the transaction scope.  sub transfer_money ($self, $from, $to, $amount) { $self-&gt;db-&gt;do_transaction(sub { if ( $from-&gt;balance &lt; $amount ) { Exception::InsufficientFunds-&gt;throw(...); } $from-&gt;dec_balance($amount); $to-&gt;inc_balance($amount); }); }  So now we check the transaction before we check the balance. Good to go, right?  No. Of course not. For most OO system I’ve worked on, that code is not good to go because objects hold their data independently from the underlying storage mechanism. What that means is that the information was read earlier and might be stale. Instead, you need to refresh your data. But you can’t just do a SELECT , you need a SELECT ... FOR UPDATE to ensure that the row is locked in the transaction.  sub transfer_money ($self, $from, $to, $amount) { $self-&gt;db-&gt;do_transaction(sub { $from-&gt;requery_for_update; $to-&gt;requery_for_update; if ( $from-&gt;balance &lt; $amount ) { Exception::InsufficientFunds-&gt;throw(...); } $from-&gt;dec_balance($amount); $to-&gt;inc_balance($amount); }); }  The above is certainly better, but my goodness, there are traps there. And it still has bugs, but I’ll stop belaboring the point now.  All you wanted to do was move a bit of money from one bank account to another. And honestly, this is a simple example. When your code gets seriously complex, it can be hard to track down bugs caused by race conditions and not having proper scope on transactions.  Which brings me to the real point of this article:   The more things you must remember, the more things you will forget.   That is something that developers often gloss over. “You have to know your system.” “It’s your fault that you weren’t paying attention.” “Be better!”  No. You want to write systems that take failure modes into account and make it hard to write those serious bugs.  Making This Simple  So, what would the above look like in Tau Station  ? Well, currently we don’t have multiple bank accounts per player, but we do have the ability to move money from your wallet to your bank account and conceptually that’s the same thing. It uses an “Exchange” system we’ve built and it looks like this:  my $exchange = $self-&gt;new_exchange( Preamble( 'deposit' =&gt; { amount =&gt; $amount } ), Steps( Wallet( $self =&gt; remove_credits =&gt; $amount ), BankAccount( $self =&gt; add_credits =&gt; $amount ), ), );  The “Preamble” is a second that declares that messages are going to be displayed to the player and what information, if any, to use for those messages. The “Steps”, however, are only what we’re actually trying to accomplish. In other words, with our exchanges, we mostly write code that describes the desired behavior. All of the “scaffolding” code is hidden away behind the scenes.  For example, note the code we didn’t have to write, but that our exchange system handles:   Exception handling  Transactions  Success messages  Error messages  Logging  Checks for whether or not we had enough money  Taxes (we have them. Long story)   And for other exchanges, we have things such as:   Job queues for asynchronous work  Email  Message popups  ... and many other details I’m omitting from this   In fact, the declarative nature of the above code means that we can even take this “exchange” and cut-n-paste it for our non-technical narrative designers and they understand what it means!  And guess what? We can reuse this. Here’s another example, reusing the BankAccount.add_credits code, for a relatively complex action of buying an item from another player (a bit simplified). But first, imagine how you would write the code for buying a physical item from a vendor and compare that code to the following.  Steps( Location( $self =&gt; is_in_station =&gt; $station ), PlayerMarketItem( $self =&gt; is_not_seller_of =&gt; $item ), PlayerMarketItem( $self =&gt; check_item_is_not_sold =&gt; $item ), BankAccount( $self =&gt; remove_credits =&gt; $amount ), BankAccount( $vendor =&gt; add_credits =&gt; $amount ), Inventory( $self =&gt; add_item_instance =&gt; $item ), PlayerMarketItem( $vendor =&gt; delete =&gt; $item ), )  Did you think of all those steps? How much code would you have had to write to implement all those steps? And would you have remembered all the possible exceptions, the transactions, the SELECT ... FOR UPDATE , and so on? Would you have remembered or cared about all the success or failure messages?  By writing code in such a declarative style, we have taken incredibly complex behavior and not only made it simpler, but more correct.  Here’s another example. In Tau Station, you can “save” your progress by gestating a new clone. If you die, you respawn into your latest close. What does clone gestation look like? It used to look like this mess:  sub gestate ( $self, $character ) { croak( … ) unless $character-&gt;area-&gt;slug eq 'clonevat'; my $price = $self-&gt;price_per_character($character); if ( $character-&gt;wallet &lt; $price ) { $character-&gt;add_message( … ); return; } my $guard = $self-&gt;result_source-&gt;schema-&gt;txn_scope_guard; $character-&gt;dec_wallet($price); $character-&gt;update; my $clone = $self-&gt;find_or_new( character_id =&gt; $character-&gt;character_id, station_area_id =&gt; $character-&gt;station_area-&gt;station_area_id, ); $clone-&gt;$_( $character-&gt;$_ ) for $character-&gt;physical_stats; my $now = DateTime-&gt;now; $clone-&gt;clone_date($now); $clone-&gt;gestation_date( $now-&gt;clone-&gt;add( seconds =&gt; $self-&gt;gestation_delay($character) ) ); $clone-&gt;update_or_insert; $character-&gt;add_message( … ); $guard-&gt;commit; return $clone; }  And that was a bucket of bugs. And hard to follow. Now it looks like this:  Steps( Location( $self =&gt; is_in_area =&gt; 'clonevat' ), Wallet( $self =&gt; pay =&gt; $price ), Clone( $self =&gt; gestate =&gt; $area ), ),  Honestly, which of those would you rather write?  Declarative Exchanges  So how does this magic work?  The Behavior  When you create a new exchange, the first thing it does is go through your steps and figure out what objects might be updated. Then we:   Start a transaction  Refresh all assets via SELECT...FOR UPDATE  Run all the steps  Commit on success and rollback on failure  Notify the player(s) (if appropriate)   It sounds simple, but there’s a lot more going on under the hood. For example, if you are a member of a syndicate and you earn income, you may have to pay “tax” to that syndicate. Thus, the exchange needs to know that it must fetch the syndicate object, lock it, and send taxes to it. As a developer writing the code, you almost never have to pay attention to this. It’s all automatic.  The Structure  Each exchange step looks very similar:  Location( $self =&gt; is_in_area =&gt; 'clonevat' )  In exhange parlance, that’s:  Topic( subject =&gt; verb =&gt; object )  Everything follows a linguistic SVO (subject-verb-object) pattern. To create a new Topic for the exchanges, you create a class called Veure::Economy::Asset:: Topic (there are legacy reasons for the name) and have it inherit from Veure::Economy::Asset . We have another system that automatically finds and loads all these classes and ensures that the asset name is exportable on demand. You just write the code, there’s no need to wire it together because that’s done for you.  Each of these classes takes a subject (the first argument) and implementing a verb is merely a matter of writing a method. The object (in linguisitic terms) becomes the argument(s) to the method. A simple is_in_area check might look like this:  sub is_in_area ( $self, $area_slug ) { my $station_area_id = $self-&gt;station_area-&gt;area_id; if ( $self-&gt;station_area-&gt;area_slug eq $area_slug ) { return $self-&gt;new_outcome( success =&gt; 1 ); } # give them a nice failure message return $self-&gt;new_outcome( success =&gt; 0, message =&gt; ... ); }  Simple, eh? And now we can reuse this to our heart’s content.  Failure  Aside from the fact that the exchanges allow us to write incredibly complex code very quickly, one of my favorite parts is the fact that even though it’s mostly declarative on the surface, underneath it’s objects all the way down. That means we get the full power of OO introspection where we need it. For example, what happens if I’m running the test suite and an exchange throws an exception?  Well, of course, we get a stack trace. And at the top of that trace, we get a stringified version of the exchange. In this case, it’s for refueling a spaceship:  character('ovid')-&gt;new_exchange( slug =&gt; 'refuel-current-ship', success_message =&gt; 'Your ship is now being refueled.', failure_message =&gt; 'Unable to refuel ship.', Steps( Area( character('ovid'), is_in, docks ), Ship( ship('ovid', 'bootlegger'), is_docked_on, tau-station ), Ship( ship('ovid', 'bootlegger'), needs_refueling ), Character( character('ovid'), not_onboard_ship, ship('ovid', 'bootlegger') ), Money( character('ovid'), pay, -15.00 ), Character( character('ovid'), refuel_ship, ship('ovid', 'bootlegger') ), Character( character('ovid'), set_cooldown, {ship =&gt; ship('ovid', 'bootlegger'), cooldown_type =&gt; 'ship_refuel',period =&gt; 1000} ), ) );  In this case, we got an exception because there’s a serious bug: somehow the character has been asked to pay negative credits. This stringified exchange shows this very clearly here:  Money( character('ovid'), pay, -15.00 ),  So it’s dead simple to recreate conditions that cause failures in incredibly complex behavior. In this case, we knew our exchange system was fine, but something was sending it bad data.  Regrets  If there is one regret I have about the exchange system, it’s that it’s not open-source. We absolutely would release this if we could, but when we started on it, it wasn’t clear what the end result would be. Thus, it’s deeply tied to our game Tau Station  . If we find ourselves with the time—or a contract—for this, we will release this for everyone.  As an aside, Ruby has something rather similar, named Trailblazer  . The main difference in exchanges is that we’re not tied to MVC or the web. Trailblazer documents itself as “A New Architecture For Rails”, suggesting a tight coupling. That being said, it has fantastic testimonials which match our internal experience with exchanges. I expect we might see more of this type of code in the future.  ","title":"Making Complex Software Simple","url":"\/articles\/making-complex-software-simple.html"},{"body":" What's a Project Manager?  The Project Management Institute defines project management as follows  :  Project management...is the application of knowledge, skills, tools, and techniques to project activities to meet the project requirements.  As a proponent of agile (in particular, WildAgile ), I find that project management is an oft-ignored area in the field. For example, Scrum defines three roles, the product owner, the team, and the ScrumMaster.   No mention of any sort of management, project or otherwise, is made. It's not that Scrum is against management; it's that Scrum is a framework for getting stuff done and management is an outside force.  So there you are, a brand new project manager (PM), or product owner (PO) and ... wait a minute. What the heck's the difference? In many agile projects, there is no project manager and the PM's responsibility is distributed across the various members of the team: product owner, developers, ScrumMaster (Scrum), Coach (XP), and so on. Thus, the definition of a PM is sometimes muddied and in my experience, varies wildly across organizations.  Curiously, when I find companies have the PO role, that&lsquo;s more clear. The product owner represents the \"voice of the customer\" and maintains the product backlog of tasks that the team works on. The Product Owner, in addition to representing the customer, also represents project stakeholders, so they have two masters, so to speak: the customers and management. But their role is the mama bird role of constantly vomiting up new tasks to feed to the baby birds of the team; they're not really project managers.  But because the waters are often muddied here, I&lsquo;m going to handwave this entire issue and hope you don&lsquo;t notice. Instead, we&lsquo;ll focus on an age-old problem in the PM\/PO space: what should we do next?  So ...  What Should We Do Next?  As a fledgling team lead, PO, PM, or whatever title your company&lsquo;s uses for person responsible for getting stuff done , figuring out what to work on next is a daunting task. It involves having an in-depth knowledge of the project, the product, the business, and the team. And when you're juggling 27 things at once, it can be maddening to have a baby bird cheeping \"what do I do next?\", and demanding a meal.  You might think that backlog grooming\/refinement  is all you need, but that still doesn't tell you, out of the 374 tasks that need to be done, which should be done next . Fortunately, there's a very simple way to do this, and that's to build a business case for each task, using just three numbers.  The Three Number Business Case  A business case is really nothing more than making an argument that a thing should or should not be done, but using numbers to back it up. If you want to learn more, I have a \"rescuing a legacy codebase\" talk that covers this.           The Three Number Problem   But what numbers do we use? For this case, try to pick the three most important numbers that represent value in your tasks.   At one point, when I had the project management role in developing Tau Station  , a free-to-play (F2P) narrative sci-fi adventure, the complexity of the role was overwhelming, but each task's \"three most important numbers\" were clear: complexity, usability, monetization.  The complexity of a task is simply how many \"story points\" (or whatever you call them) that team members have assigned to a task. I recommend using Fibonacci numbers for story point estimates  , especially for what I'm about to show you.  The usability of a task was how useful it was to the customers. Again, you want to use Fibonacci numbers here. You need a feel for what this number is because, in reality, you cannot get a perfect estimate. However, it's clear that a feature which lets customers download reports is more valuable to them than a feature which slightly adjusts the color of a sidebar, so the former might get the number \"13\" while the latter might get the number \"1\".  In the case of Tau Station, everyone is allowed to play the game for free, but as with many F2P games, there are monetization options in the game   and anything which increases the likelihood of monetization gets a higher number. A \"Happy Hour\" feature, where the players get extra rewards when they buy stuff gets a higher monetization number than a feature to make forum posts \"sticky.\" Again, you're using Fibonacci numbers here.  Important: the three things you choose should reflect your business concerns and may very well be vastly different business requirements. You may choose \"security\" or \"legal compliance\" or something else entirely. Tau Station included these as part of the \"usability\" number. Think hard about these numbers because you don't want to get them wrong.  Once you have your three numbers, what next? Well, you need to assign weights to them. This will largely be a process of trial and error until it feels right. In our case, let's say that both monetization and complexity get a weight of 40 and usability gets a 20. Then you play around with formulae until you get a final score, with higher-scoring tasks having a greater priority. Here's the formula I used for Tau Station, though you will need to adjust this for your own needs:  pros = monetization * 40 + usability * 20 cons = complexity * 40 scale = 10 score = ( ( pros – cons ) + 500 ) \/ scale  Let's say a task has a monetization of 1, and a usability of 5, and a complexity of 7. How does that compare with a take of a monetization of 3, a usability of 3, but a complexity of 13? The first task might earn us less money and be less useful, but it's easy to implement. Decisions, decisions ...  Well, monetization and usability are both \"pros\" (benefits), and complexity is a \"con\" (drawback), so we have task 1:  pros = 1 * 40 + 5 * 20 cons = 7 * 40 scale = 10 score = ( ( 140 – 280 ) + 500 ) \/ 10  And task 2:  pros = 3 * 40 + 3 * 40 cons = 13 * 40 scale = 10 score = ( ( 240 – 520 ) + 500 ) \/ 10  Task 1 has a final score of 36, while task 2 has a final score of 22. Thus, we do task 1 first.  Once you learn to plug those three numbers into a spreadsheet, you can then sort the tasks by score and prioritize tasks at the top of the list.    A Spreadsheet Showing Our Three Number Business Case   Real World Usage  When I was doing this, I found it made my work much easier and the team appreciated it because it gave a clear direction. Plus, since the team provides the complexity estimates, they know that they have a real impact on project direction, something that teams usually appreciate.  I used an Excel spreadsheet, but also a small program that I wrote which fetched data from github and ZenHub  to automate the creation of my spreadsheet. Every month I'd repeat the process, fill in missing numbers, sort the rows and share it with the team via a Google Doc. It worked very well and figuring out the project tasks for the month was no longer a stressful affair..  Note that sometimes you will find tasks which don't seem to be scored correctly with this technique. When that happens, investigate whether or not your three numbers seem reasonable. Maybe your weights are wrong, or your formula is off.  Or, as sometimes happens, there are external factors. A task may be complex, offer low usability, and no monetization potential, but another team is demanding that XML feed now and hey, you have to play nice with others. When odd events like that happen, use your judgement instead of the spreadsheet.  At the end of the day, this technique, like all others, isn't perfect. When used correctly, however, it's a great tool for building value and focusing on your real needs. Further, you prioritize greater value and, when asked to justify your decisions, you have numbers to back them up. Even if they're not perfect, quibbling over numbers and formulae instead of hunches is a much safer career move.  ","title":"Project Management in Three Numbers","url":"\/articles\/project-management-in-three-numbers.html"}]
\ No newline at end of file
diff --git a/root/blog/marc-andreessen-techno-babble.tt2markdown b/root/blog/marc-andreessen-techno-babble.tt2markdown
index 8b64b0ac..2769d83e 100644
--- a/root/blog/marc-andreessen-techno-babble.tt2markdown
+++ b/root/blog/marc-andreessen-techno-babble.tt2markdown
@@ -28,7 +28,7 @@ straw men, false dilemmas, and reads like a high school sophomore barely
 squeaked out a passing grade in a macroeconomics course.
 
 Were it not for the fact that it's authored by a famous billionaire, it would
-just be another screed on another blog no one reads.
+just be another screed on another blog no one reads (like this one).
 
 # Accelerationism
 
diff --git a/static/js/search/tinysearch_engine_bg.wasm b/static/js/search/tinysearch_engine_bg.wasm
index 5e8fc883..41472830 100644
Binary files a/static/js/search/tinysearch_engine_bg.wasm and b/static/js/search/tinysearch_engine_bg.wasm differ