Friday, June 28, 2019

Red Hat Summit 2019: My notes

My notes from sessions at Red Hat Summit 2019 are for my reference and as documentation for any submitted continuing education credits.

The Ansible party was awesome as usual (even if was a part of the Smart Management party). Great food at Legal Harborside with lots of people I wanted to see.

I'm glad I watched most keynotes remotely. The one I did attend in person reminded me of how cold that space is and how many people where chemical scents that trigger my asthma.

Ran into more cool people at the Red Hat Women’s Leadership Community Luncheon.

Remaining notes include sessions attended as a reminder of which slides or videos to reference for more details as well as topics, commands, and keyword to dig into in the future.

Keynote recordings are available in the YouTube channel.

Session descriptions have links (where available) to slide decks.

On Demand session recordings require a login.

5/7: Red Hat security roadmap : It's a lifestyle, not a product

  • Speaker: Mark Thacker, Red Hat
  • Slides available
  • Recording available

5/7: The current and future state of security: A discussion of security challenges (Birds of a feather)

5/7: Successfully implementing DevSecOps: Lessons learned

  • Speakers: William Henry, Red Hat; Deven Phillips, Red Hat, Inc.; Lucy Kerner, Red Hat
  • UBI - Universal Base Image
  • Case Study: Homeland Security in Innovative Labs
  • Look at pipeline box on Heritage slide.

5/7: Security: Emerging technologies and open source

  • Speaker: Mike Bursell, Red Hat; Nathaniel McCallum
  • Slides available
  • Recording available

5/8: Top 10 security changes in Red Hat Enterprise Linux 8

  • Speaker: Mark Thacker, Red Hat
  • Slides Available
  • Recording Available

5/8: Security and compliance automation: Demos of current capabilities and future technologies

  • Speakers: Shawn Wells; Chris Reynolds, Red Hat; Gabriel Alford, Red Hat Inc
  • Included pipelines with Ansible Tower, SCAP, and Open Controls.

5/9: Red Hat on Red Hat: Transitioning Red Hat IT to hybrid cloud infrastructure using OpenStack and Ceph Storage

  • Speakers: Brian Atkisson, Red Hat, Inc.; Matthew Carpenter, Red Hat, Inc.
  • Slides Available

5/9: Evolution of a Linux system identity and authentication stack

  • Speaker: Dmitri Pal, Red Hat, Inc.
  • Slides Available

5/9: A practical introduction to container security using CRI-O (LAB)


Tuesday, March 5, 2019

20 Years Ago: Remembering my first RHCE exam

When, where, and why

I learned about network operating systems working in a person computer (PC) helpcenter. After several years talking to customers and being an escalation point for other support engineers, I moved into the training department.

I was responsible for training the technicians using both in house written materials for our own hardware products and partner materials for most of the software. When I started working with Linux, I was already teaching official material for OS/2 Warp Server and SCO Unix in additional to some material for Microsoft, Banyan Systems, and Novell products.

When IBM invested in Linux, my position in a training department and my Unix background made me a lead on the team facilitating a plan to get the PC Helpcenters around the world up to speed in supporting four Linux distributions. We needed to quickly ramp up on Red Hat, Caldera, SUSE, and TurboLinux. I became the point person for Train-the-Trainer sessions on all four distributions in addition to having the responsibility of getting the North America support center trained.

The Red Hat office, with their brand new Training and Certification team, was located halfway between my office and my home so of course I started by attending their Red Hat Certified Engineer course. That was 20 years ago. To be precise, it was Mar 1-5, 1999.

The course

The course was similar to other technical training course which I had attended and taught. The RH300 RHCE Course was new and it was the only course offered at the time. In those years, the authors and the instructors were the same people.

It was a lot of information for a single week and it required a good foundation of prerequisite knowledge. Some things have not changed in 20 years!

The exam

I could not find the course or exam descriptions from 1999 but the Internet Archives Way Back Machine does show the spring of 2000 Prep Guide:

My memory is that most items did not change much in that first year.

The format:

Like now, the the exam was on that Friday. It was mostly hands on but not entirely since the first iteration included a multiple choice section. It was three parts:

  • Installation Lab Exam - 2.5 hours
  • Written Exam - 1 hour
  • Debug Lab Exam - 2.5 hours
  • PASS requires: avg of 80 or higher, with no single score lower than 50 pts.

I think my class did the installation in the morning with debug in the afternoon. The written section was in between followed by a lunch break. Soon after, the lab sections were swapped out in the daily schedule. I suspect it was quicker to grade the debug than the installation lab and systems needed to be reset for each section. My class did not need a reset since we took the exam on zip drives. Yup, you heard that correctly, ZIP drives. They were removed, labeled, and graded later. The email with my score report is dated March 19th. Two weeks later!

The objectives:

Objectives that are still seen today included:

  • Red Hat installation and network configuration
  • filesystem layouts and user management
  • boot issues and boot loader options
  • package management and automated installation
  • various network service configuration and security

Of course at that time, it was network-scripts files, ext2, NIS, LILO, rpm, squid, tcp_wrappers, and ipchains instead of the nmcli, xfs, GRUB, yum, systemd, and firewalld.

There were also some "get off my lawn" objectives:

  • We had to understand XFree86 configuration and get graphical login managers working
  • We used boot floppies for rescue. Yes, Floppies!
  • And we had to "be able to configure, build, and install the Linux kernel and modules from source".

The version:

  • My exam in March 1999 was on Red Hat Linux 5.2.
  • GNOME was technology preview and the kernel was 2.0.36.
  • A month later Red Hat Linux 6.0 was released.
  • The 2.2 kernel and glibc 2.1 were major new features and GNOME was the default GUI.

What Came Next

I became a Red Hat Certified Instructor through a partner program and immediately started sharing the wonders of Open Source with a whole new set of users. I added Red Hat Certified Examiner credentials a year later.

The first class I taught using Red Hat Training materials was written for RHL 6.0 and delivered in Scotland. That, though, is a story for another article.

Friday, March 1, 2019

FeBRRRRuary reading and writing

Stuff I wrote:

Getting started with Vim visual mode at turned out to be very popular.

Fedora IoT Docs are Live for the Fedora Community Blog is a summary report for the majority of my February writing.

A technical reboot, recharge, upgrade, and expansion of the Fedora IoT Documentation took up most of my time and provided an opportunity to spend the dreary month working from home instead of commuting.

New bookmarks:

Specifically for the Fedora IoT project, I did a lot a reading in the month as well. Here are a few of the items I bookmarked. Some inspired my hacking and writing, some are saved for future adventures:

Getting to Know Fedora Silverblue

Raspberry Pi improvements in Fedora 29

Fedora IoT with Peter Robinson | OpenHours ep 133 video from 96Boards Open Hours.

How to turn on an LED with Fedora IoT

Turn a Raspberry Pi 3B+ into a PriTunl VPN

Set the holiday mood with your Raspberry Pi

How to build fully automated musical lights [Halloween/Christmas]

Home Assistant Installation on Docker.

Mozilla IoT Gateway

Sunday, January 14, 2018

Prime - not impressed

I'm a Whole Foods junkie. Or was, depending on if you still consider it Whole Foods or if you already call it Amazon Foods. It started before Whole Foods really with family and food allergies but as my allergies got worse it has become a life saver when traveling. I knew I had a place where the staff was trained to answer questions about the ingredients and most things are very well labeled. People have said to me, "Oh we have a Wegmans" or "we have a Trader Joes" and these chains are useful in frozen dinners or raw ingredients and they do have prepared food. I have check out other local recommendations too as I have traveled but so far no one has done as well as Whole Foods in having the labeling and variety for picky (choice or allergy) eaters like me.

Sure, they have specially items, many of which are expensive and only for the privileged. And prepared food is more expensive anywhere but even fancy organic Whole Foods hot bar is less than most room service when traveling. Also, when you eat all clean food, all the time, you actually can get a lot more nutrients and that full feeling with a lot less quantity. My family has tracked the budget.

We use a lot of the 365 Brand items but mostly we cook simple meals from scratch so we have a lot of fruit and veggies.  We have a garden, we shop at the farmers market, and we supplement from the local coop and from the Whole Foods.

All this is background leading to the my skepticism with the Amazon acquisition of Whole Foods last year. While hoping that the larger buyer could make better deals in price and worrying about what it will mean for the employees (who appeared to be getting a fair wage and like working in the store so far) in the future, I also wondered (still wonder) what it will mean for me.  So far I have not been impressed. The items with lower prices have also been lower quality and the variety of products, especially in the category of more expensive but tastes great allergen friendly ones, is slowly dwindling.

Once Whole Foods + Amazon was up and running, I looked to see about getting some items delivered straight to the house. Most of the items I want are in Prime Pantry only so you have to have Prime which I didn't have it. Yet. Amazon is also beginning to offer some deals in the store too if you have Prime such as a substantial discount per pound on a Fresh Turkey at Thanksgiving.

So I was willing to check it out. I had three things to investigate with Prime - shipping in general, prime pantry, streaming videos. So I set up the free trial.

I knew that starting at the end of the holiday shopping season would have its own issues but I wanted to see how it worked in my area.  I live in a rural area with the mailbox almost a mile from house so I use a PO box in town for most mailings. I hoped that the Prime 2-day shipping would use carriers that delivered straight to the house. Unfortunately, Amazon uses a lot of USPS and if the package is small enough to fit in the box (and we have a package box out there too) it gets left way away from the house. I might as well continue with the shipping to the PO box so items are secure and dry. I rarely need something right away so waiting to order when I have enough for standard free shipping works for me.

So what about streaming videos? I found a few things not in Netflix that I might watch again but mostly they don't have anything better and I find myself back in Netflix most times. I'll take advantage of it if I have it, but I don't see any value in getting Prime solely for this feature.

Prime Pantry is interesting and if I did not live close enough for a weekly drive to an actual Whole Foods store, I might consider it. The first time I looked, they didn't have much. The next time I looked, there were enough items for me to fill a box if I needed them. I also checked and yes, the prices are exactly the same online as in the store. Then last week I decided to actually give it one try before my trial ends but 9/10 items I would get were out of stock and unknown availability.  That rules that out. I was already leaning to a no. It is $6 per box for delivery on top of the monthly (or yearly) fees for Prime. I can drive about 20 miles to either of the nearest Whole Foods for less and find more of the items on my list plus all my other groceries.

I do not live in an area of Prime same day shipping -  though I am only 25ish miles from a fulfillment center - so 2 hour or Prime Fresh deliveries are also completely out and not even on the list possibilities anytime soon. If I lived in one of those areas or had a larger family to order items for or just shopped online more, it might be worth it. But so far, for where I live and what I buy, I am not impressed.

The cancellation requires what seems like a gazillion times confirming you want to cancel as they remind you of all the benefits of staying but I finally got through it and have ended my Prime trial.


Thursday, January 11, 2018

InfoSec Basketball Rebounds

I was reading a post about red team vs blue team and all the support for purple teams and was struck with inspiration from one of the comments mentioned: 

"(offense wins games, defense wins championships)"

They do not appear to be implying basketball and given the timing of the comment, they may have been thinking more about football.  For me though, as a huge fan of Women's Basketball, I recognized it as a variation on a quote from the great Pat Summit:

"Offense sells tickets, defense wins games, rebounding wins championships."

And suddenly a hobby and work collide. My brilliant inspiration comes from how this really does apply to information security as well.

Everyone loves good offense. For some it is high flying dunk or a buzzer beater from half court. For others it is a successful, innovative attack as part of a red team. We attend "ethical hacking" courses because breaking into things is fun. Big exploits make the news and get cute little logos. Other crackers just keep working and getting two points here and there until it adds up on the scoreboard.

Defense is what is needed to win the game though. It doesn't matter how many point you put up if the other team is allowed to put up more. Can you prevent the problems in the first place? Have you done the basics of password security and patch management? Are we monitoring the logs? And even if it is done well, it might not make a stat line. Sure a couple blocks and few steals here and there but the standard box score doesn't list shot clock violations and a zone defense rarely makes the sports center top ten. The blue teams that prevent attacks from being expensive do not get badges (logos).

Shutouts are very rare and never in championship matches. Defense will not stop everything and even the best offense misses a lot of shots. Rebounds are how we react on a miss. Defensive rebounds end an opportunity for more points. The offense got a shot off, but if they miss, did you stand and watch or did you go after the ball and box out the opponent? If you are on offense and your shot bounces back, your goal is to secure the ball and try again. Attack another way, find another opening, maybe the same opening if you didn't get boxed out. With information security, are you monitoring logs, are your alerts set up correctly, are you reacting to even the missed attempt or are you just waiting and letting them take another shot? Are you boxing out?

Offense often comes down to skill and on the court, natural ability plays a big part. Defense can be taught with basic and repetitive drills. Rebounding is about heart. You don't have to be the tallest or biggest or strongest. Who wants it more? Who will go after the ball? Who can read the play and be the in correct position to respond? 

And no one wins anything without a contributions in all areas and a whole lot of teamwork.

When it comes to programming and coming up with creative attacks, I do not have the natural abilities to make a good red team member. I am much more comfortable practicing defense and jumping into position to grab a rebound.  Along with some post game analysis and armchair coaching! 


Wednesday, January 10, 2018

Watching the meltdown.

I have been watching Meltdown and Spectre unfold from the sidelines. Other than applying available updates, I'm just watching and absorbing the process of the disclosure. This one appears mid way along a long road.

I teach mostly administrators. I teach some developers. I teach those in, or desiring to be in, infosec. I like teaching security topics. I think securing systems requires more people thinking about security from the beginning of design and as an everyday, no big deal part of life. A question I ask with these newsworthy issues is what normal practices can mitigate even part of the problems?  There are two big basics - least privilege and patch management - to always keep in mind. Issues like ShellShock and Venom were mostly mitigated from the beginning with SElinux enabled (least privilege) and WannaCry had little impact on those systems patched long ago when the SMB bug was first found and fixed.

However, in some cases, both exploits and accidents come from doing something that no one else thought of trying. This is why I like open source. There is the option (not always used) for more people trying different things and finding better uses as well as potential flaws. Any type of cooperation and collaboration can be the source of some of these findings including pull requests, conference talks, or corporations working with academic research projects.

Spectra and Meltdown are not the first bug of their kind, nor the last. Anything that grabs or holds more information than is requested - such as cache or speculation - is bound to eventually grab and expose something it shouldn't. Or allow some type of injection. I gave some kudos to the team getting the credit for this discovery and got some push back from a friend defending another friend that gave a related talk at a conference in 2016. Maybe not enough credit is given to those that speculated (pun intended) on this type of problem in the past. This timeline lists several and some retweets from people I trust to be smarter than me in this topic point to ideas even older.

The Google Project Zero team is getting the recognition because of a variety of pieces in a big puzzle. Right place, right time. Privilege from the backing of a large company. Their use of the embargo and disclosure process working across the industry. A new proof of concept and published paper. Indications of ways to exploit it at scale. A mitigation. It all comes together and suddenly more than just the researchers realize the scope of the risk that has been taken. Intel is getting more than their share of the blame too when people recognize a company name faster than a general concept or part of a computer. And, yes, in some cases there is also too much fluff and fear in the reporting.

The embargo and disclosure process is pretty interesting too. I sat in a talk a couple of years ago about how a large company deals with this in the open source world and Mike Bursell has a post with thoughts about it again in reference to this case. I actually had an idea something big was coming from the combination of noise and speculation about patches being submitted and who was NOT talking about them.

We are still discovering the full impact of the CPU design decisions made. Sure, they are serious, especially as more people are able to automate attacks against the vulnerability, but they are also nothing to panic about. This is not just an Intel problem. It is a market driven quest for more power with less money and despite various risks. We are all to blame. Apply the patches, monitor the impact, invest in the next generation of inventors and inventions. In other words, business as usual.

The choices were made in favor of optimization, so will things be a little slower now? Probably for many people, but not everyone. Will we get over it? I would think so.

What will happen in the long run with the latest news? I predict many people will choose performance over security. I predict a few years from now when someone finds a scalable way to exploit one or more of the variations, people will have forgotten that they should have updated bios, firmware, and kernels today. If we are lucky, they will have the latest patches already deployed and just need to make some configuration changes. But when has luck worked out as the best security practice?

Links I have collected helping me to understand:

SANS Institute webcast.

Fedora Magazine KTPI overview.

OpenStack, What you need to know.

Project Zero technical overview.


My favorite analogy thread - the library comparison - (more were rounded up here).


Tuesday, January 9, 2018

A cold start

Weather dominated a large part of my local world for the start of 2018.

In both the duration of the cold:

And the low temperatures (I saw -1 F at my place! Brrrrr!!!!):

Central NC doesn't usually get cold enough long enough for much running water to freeze. And yet:

The little snow we got was very pretty.

I like living far enough north that we get this once or twice a year and far enough south that we can usually wait out the melting. This one took a lot longer to melt though!

Of course, since we do not usually have this kind of weather, it also comes with all its problems. This area gets laughed at for cancelling school on a forecast, but the normal infrequency means only enough equipment for clearing emergency routes and making sure kids are not stranded at school overnight. There were school delays and cancellations from the cold that overshadowed the one day where the cancellation were more about the snow. When you rarely need them, you generally also don't have engine warmers for ALL the buses that need to be started in the mornings. And there are some long cold routes to cover as well.

I did have to get the furnace fixed but with good insulation, a space heater, and the neighbors, I easily survived 24 hours with no heat in the house.  I did fine with water but I know people with frozen well heads and pipes. The towns were also kept hopping to fix water main breaks. Ours just don't make the news like JFK Airport! I got heat back just before the snow arrived and was lucky enough to not loose power (and heat) that night when transformers blew. The picture posted was beautiful but I would have been willing to miss out on that in exchange for a quiet and warm night for all.

And now that things are beginning to thaw, we will find the next round of issues.

I have friends and family in the Boston area so I was keeping an eye on the big storm there too as well as the weather near family in NH.  We are getting an early start to the total cost of weather disasters for the year (NPR just reported on 2017).

Today is warm(ish) and sunny. A good day for an outside walk. Bring on the rest of the year!