Introduction¶

Ubuntu 22.04 removes a native Firefox package in favor of a snap package. I'm sure this has advantages.

But the reality for me was several fold: startup times were noticeably slower, and the selenium geckodriver just plain didn't work for me (issue here), with some debate online but no canonical solution. I also couldn't get Jupyterlab to autolaunch (minor, but annoying).

Solution below reproduced from https://balintreczey.hu/blog/firefox-on-ubuntu-22-04-from-deb-not-from-snap/ with adaptations which worked for me.

Solution¶

You can still install Firefox as a native deb from the Mozilla team PPA. The process which worked for me was:

Step 1¶

Add the (Ubuntu) Mozilla team PPA to your list of software sources by running the following command in the same Terminal window:

sudo add-apt-repository ppa:mozillateam/ppa

Step 2¶

Pin the Firefox package

echo '
Package: *
Pin: release o=LP-PPA-mozillateam
Pin-Priority: 1001
' | sudo tee /etc/apt/preferences.d/mozilla-firefox

Step 3¶

Ensure upgrades will work automatically

echo 'Unattended-Upgrade::Allowed-Origins:: "LP-PPA-mozillateam:${distro_codename}";' | sudo tee /etc/apt/apt.conf.d/51unattended-upgrades-firefox

Step 4¶

Install firefox (this will warn of a downgade - ignore it)

sudo apt install firefox

Step 5¶

Remove the Firefox snap

sudo snap remove firefox

Conclusion¶

This worked for me - Firefox starts, my existing Selenium scripts work.

Situation¶

You do this:

$ git clone https://some.site/git/some.repo.git
$ cd some.repo
$ npm install

Pretty common right? What can go wrong?

What about this:

curl -L https://our-new-thing.xyz/install | bash

This looks a little unsafe. Who would recommend it? Well it's still one of the ways to install pip in unfamiliar environments. Or Rust.

Now installing from these places is safe: why? Because they're trusted. There's huge reputational defense going on. But the reality is that for a lot of tools - npm being a big offender, pip too - there's all sorts of ways that while sudo and user permissions will protect your system from going down, your data - $HOME and the like - basically all the important things on your system - are exposed.

This is key: you are always running as "superuser" of your data. In fact your entire operating environment - systemctl --user provides a very useful and complete way to schedule tasks and persistent daemons for your entire user session. There's a lot of power and persistence there.

Problem¶

There's two competing demands here: it's pretty easy to build isolated environments when you feel like you're under attac, but it takes time - time you don't really want to commit to the problem. It's inconvenient - which is basically the currency we trade when it comes to security.

But the convenience<->security exchange rate is not fixed. It has a floor price, but if we can build more convenient tools, then we can protect ourselves against some threats for almost no cost.

Goals¶

What we want to do is find a safe way to do something like npm install and not be damaged by anything which might get run by it. For our purposes, damage is data destruction or corruption beyond a sensible scope.

We also want this to light weight: this should be a momentary "that looks unsafe" sort of intervention, not "let me plan out by secure dev environment".

Enter Bubblewrap¶

bubblewrap is intended to be an unprivileged containers sandboxing tool and has as its specific goal the elimination of container escape CVEs. It's also just available in the Ubuntu repositories which makes things a lot easier.

This is a fairly low level tool, so let's just cut to the wrapper script usage:

#!/bin/bash
# Wrap an executable in a container and limit writes to the current directory only.
# This system does not attempt to limit access to system files, but it does limit writes.

# See: https://stackoverflow.com/questions/59895/how-to-get-the-source-directory-of-a-bash-script-from-within-the-script-itself
# Note: you can't refactor this out: its at the top of every script so the scripts can find their includes.
SOURCE="${BASH_SOURCE[0]}"
while [ -h "$SOURCE" ]; do # resolve $SOURCE until the file is no longer a symlink
  DIR="$( cd -P "$( dirname "$SOURCE" )" >/dev/null 2>&1 && pwd )"
  SOURCE="$(readlink "$SOURCE")"
  [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
done
SCRIPT_DIR="$( cd -P "$( dirname "$SOURCE" )" >/dev/null 2>&1 && pwd )"

function log() {
  echo "$*" 1>&2
}

function fatal() {
  echo "$*" 1>&2
  exit 1
}

start_dir="$(pwd)"

bwrap="$(command -v bwrap)"
if [ ! -x "$bwrap" ]; then
    fatal "bubblewrap is not installed. Try running: apt install bubblewrap"
fi

export PS_TAG="$(tput setaf 14)[safe]$(tput sgr0) "

exec "$bwrap" \
    --die-with-parent \
    --tmpfs / \
    --dev /dev \
    --proc /proc \
    --tmpfs /run \
    --mqueue /dev/mqueue \
    --dir /tmp \
    --unshare-all \
    --share-net \
    --ro-bind /bin /bin \
    --ro-bind /etc /etc \
    --ro-bind /run/resolvconf/resolv.conf /run/resolvconf/resolv.conf \
    --ro-bind /lib /lib \
    --ro-bind /lib32 /lib32 \
    --ro-bind /libx32 /libx32 \
    --ro-bind /lib64 /lib64 \
    --ro-bind /opt /opt \
    --ro-bind /sbin /sbin \
    --ro-bind /srv /srv \
    --ro-bind /sys /sys \
    --ro-bind /usr /usr \
    --ro-bind /var /var \
    --ro-bind /home /home \
    --bind "${HOME}/.npm" "${HOME}/.npm" \
    --bind "${HOME}/.cache" "${HOME}/.cache" \
    --bind "${start_dir}" "${start_dir}" \
    -- \
    "$@"

In addition to this script, I also have this in my .bashrc file to get nice shell prompts if I spawn a shell with it:

if [ ! -z "$PS_TAG" ]; then
  export PS1="${PS_TAG}${PS1}"
fi

The basic structure of this invocation is that the resultant container has networking, and my full operating environment in it...just not write access to any files beyond the current user directory.

This is a handy safety feature for reasons beyond a malicious NPM package - I've known more then one colleague to wipe out their home directory writing make clean directives.

Usage¶

Usage could not be simpler. With the script in my PATH under the name saferun, I can isolate any command or script I'm about to run to only be able to write to the current directory with: saferun ./some-shady-command

I can also launch a protected session with saferun bash which gives me a prompt like:

[safe] $ 

This is about as low overhead as I can imagine for providing basic filesystem protection.

Conclusions¶

This is not bullet-proof armor. And it certainly won't keep nosy code from poking around the rest of the filesystem. Are you 100% confident you never saved an important password to some file? I'm not. But I do normally work with a lot auxillary commands and functions around my home directory, and I like them being mostly available when doing risky things. This strikes a good balance - at the very least it limits the damage scope of running some random script you downloaded from causing real nuisance.

I recommend checking out bubblewrap's full set of features to figure out what it can really do, but for something I knocked up by reading for a few hours this added a handy tool to my repository for me.

The Problem¶

So I've had a Generation 2 Toyota Prius since 2004. Coming up on 17 years old now in Australia, and recently I finally had what turns out to be the dreaded PA080 fault code get thrown - this is a general hybrid traction battery error.

Since the battery is relatively expensive compared to the value of the car and I don't like spending money anyway, the question becomes what can we do about this?

DIY Reconditioning¶

Fortunately, the car is old enough that this problem has happened before. Over at https//priuschat.com and elsewhere on the web, people have disassembled then Prius traction battery and fixd this problem themselves.

There are basically 2 issues at play: general NiMH degration, and polarity reversal - cell failure.

Cell Failure¶

In general, the PA080 code (at least in my experience), happens when a battery module will suddenly drop its voltage by over 1V.

This happens due to a phenomenon in NiMH cells called "polarity reversal" - characterized by a discharge curve like this one:

Source

It is what it sounds like: under extreme discharge conditions, the NiMH cell will go to 0, and if left in this state for too long (or in a battery pack where current continues to be pulled through the cell) it will then enter polarity reveral - positive becomes negative, negative becomes positive. This is disasterous in a normal application, and devastating in a battery pack as the cell now gets driven in this condition by regular charging to continue soaking up current producing heat.

At this point, the cell is dead. In a Prius battery module of 6 cells, a reduction in voltage of about 1V means you know you've had a cell drop into reverse polarity and its not coming back.

NiMH battery cells primer¶

It's important to understand NiMH cells to understand why "battery reconditioning" is possible and advisable.

Source

Standard NiMH battery chemistry has a nominal voltages of 1.2V. This has little bearing on the real voltages you see with the cells - a fully charged cell goes up to 1.5V, considered to be the absolute top and you're evolving hydrogen at that point - and a single, standalone cell, can be take all the way to 0V (this is not safe - miss the mark and you wind up in polarity reversal).

In a battery pack of NiMH cells, these lower limits are higher for safety: pack cells all have slightly different capacities, and once you hit 0V on one, if the others don't hit 0V at the exact same time then the empty ones will get driven into polarity reversal. At roughly 0.8V you start running into a cliff of voltage decay anyway, so that's generally the stopping point.

The graph below is an excellent primer on the voltage behaviors of NiMH at different states of charge. Note that the nominal voltage is measured right before the cell is practically empty, but for most of its duration voltage is very constant - almost linear - until the cell is almost full.

Source

$$\require{mhchem}$$

Degradation Mechanisms¶

The above explains the behavior of NiMH cells, but not why we can recondition them in a vehicle like the Prius. To understand this, we need to understand the common NiMH battery degradation mechanisms.

NiMH chemistry is based on the following 2 chemical reactions:

Anode: $\ce{H2O + M + e^- <=> OH^- + MH}$

Cathode: $\ce{Ni(OH)2 + OH^- <=> NiO(OH) + H2O + e^-}$

Note the M: this is an intermetallic compound, rather then any specific metal is essentially where a lot of the R&D in NiMH batteries goes.

Our target of recovery is the cathodic reaction involving the Nickel. In normal operation the Prius runs the NiMH batterys between 20-80% of their rated capacity. This is, in general, the correct answer - deep discharging batteries causes degradation of the electrode materials which is a permanent killer (over the order of 500-1000 cycles though).

Crystal Formation¶

The problem enters with an issue known as "crystal formation" when the batteries are operated in this way over an extended period. Search around and you'll see this referenced a lot without a lot of explanation and mostly in context of Nickel-Cadmium (NiCd) batteries.

NiMH's were meant to, and were a huge improvement on, most of the "memory effect" degradation mechanisms of NiCd batteries, however some of the fundamental mechanisms involved still apply as they are still based on the same basic active materials on the cathode - the Nickel Hydroxide and Nickel oxide hydroxide.

There are many, many mechanisms of permanent and transient change in NiMH batteries, but there are 2 identified which can be treated by the deep charge-discharge cycle recommended for reconditioning.

One is that observed by Sato et. al.: nickel oxide hydroxide has 2 primary crystal structures when used in batteries - β‐NiOOH and γ‐NiOOH.

β‐NiOOH and γ‐NiOOH are generally recognized as being two in-flux crystal states of the Nickel electrodes of any nickel based battery with a (simplified) schema looking like the following:

Source

γ‐NiOOH is the bulkier crystal form, and has more resistance to hydrogen ion diffusion - this is important because the overall ability of the battery to be recharged is entirely dependent on the accessibility of the surface to $\ce{H^+}$ ions to convert it back to $\ce{Ni(OH)2}$.

What Sato et. al. observes is that during shallow discharging and overcharging of NiCd cells, they see a voltage depression effect correllated with a rise in γ‐NiOOH peaks on XRD spectra. When they fully cycled the cells, the peaks disappeared - the γ‐NiOOH crystals over several cycles are dissolved back to $\ce{Ni(OH)2}$ during the recharge cycle.

| SEM photographs captured at 10 μm of the positive plates of (a) a good battery, (b) an aged battery, and (c) a restored battery. Note: these were NiCd's, but a similar process applies to the nickel electrode of an NiMH cell. Source |

Although the Prius works hard to avoid this sort of environment - i.e. the battery is never overcharged - it's worth remembering that the battery is not overcharged in aggregate - but it's a physical system, with a physical environment. Ions need to move around in solution, and so while in aggregate you can avoid ever overcharging a cell - on a microsopic levels through random change every now and again an overcharge-like condition can manifest. That said - it took my car 17 years to get to this point.

There's more detail to this story - a lot more - and pulling a complete picture out of the literature is tricky. For example the γ‐NiOOH phase isn't considered true γ‐NiOOH but rather γ'‐NiOOH - the product of Nickel intercalating into γ‐NiOOH, rather then potassium ions (from the potassium - $\ce{K^+}$ used as electrolyte in the cell). It's also a product of rest time on the battery - the phase grows when the battery is resting in a partly charged state.

The punchline of all of this is the reason Prius battery reconditioning works though: the Prius is exceptionally good at managing its NiMH cells, and mostly fights known memory effects while driving. However, it can't fight them all the time and with time and age you wind up with capacity degradation due to crystal formation in this ~50% state-of-charge (SOC) range. And importantly: it's experimentally shown that several normal cycles is highly effective at restoring it by dissolving away the unwanted phase.

Dehydration¶

There's a secondary degradation mechanism that's worth noting for those who have seemingly unrecoverable cells in a Prius: dehydration.

Looking again at the NiMH battery chemistry -

Anode: $\ce{H2O + M + e^- <=> OH^- + MH}$

Cathode: $\ce{Ni(OH)2 + OH^- <=> NiO(OH) + H2O + e^-}$

you can see that water - $\ce{H2O}$ - is involved but not consumed in the reactions. This is also kind of transparently obvious: you need an electrolyte for ion exchange. What is not obvious though is that the situation under battery charging is technically a competitive with a straight electrolytic water-splitting reaction:

$\ce{2H2O <=> 2H^2 + O^2}$

This is a known problem - though largely resolved from normal recombinative processes in the battery (having a shared gas headspace allows the H2 and O2 to recombine back into water) and can be assisted by adding specific recombination chemistry and normally just resembles a loss function on charging the cells, simply producing heat.

This is a tradeoff in battery design: a sealed cell doesn't leak gas, which ensures it can eventually recombine. But a sealed cell can overpressure and rupture, at which point the cell is destroyed. The Prius cells are not sealed - a one-way overpressure blow off valve is present which vents at 80-120 psi - 550-828 kPa (this is substantial) - and the cells themselves depend on being clamped to prevent gas pressure from damaging them during charging.

But the result is the same: failed seals or overheated cells over a long duration may have lost water through either electrolysis processes.

There are ways to fix this sort of failure - and the results are spectacular - but this is definitely into "last resort for experimentalists" sort of intervention. Typical NiMH design uses a 20-40% w/v KOH solution in water. LiOH is added to improve low temperature performance, and NaOH is substituted partially or fully for reduced corrosion in high temperature applications.

Per this link 30% w/v KOH and 1.5 g/L LiOH is suggested. For the purposes of cell rehydration, an exact match is probably not important as a "dried out cell" will still contain all its salt components (though depending on redissolving them may not be the best option). A starting point for other mixes might be this paper which concludes a 6M KOH solution is optimal.

The big results reported over by this PriusChat member for anyone considering this are here - where he notes he used 20% KOH. Of note: getting deionized water, and a suitably un-metal contaminated salt, is probably key to success here (as well as sealing up the cells properly - the trickiest part by all accounts). That said - various metal dopants are used in NiMH cells to contribute all sorts of properties, so this may be a small effect. It is worth worrying about polymeric impurities in salts - you can eliminate these by "roasting" the salt to turn the into carbon ash.

It is noted in the literature that 6-8M KOH is the sweet spot for discharge capacity - however the use of a 1M solution for total cycle life has also been noted here.

One key parameter for anyone considering this is a rule of thumb figure for electrolyte volume of 1.5 - 2.5 mL A/h. For Prius cells this corresponds to 9.75 - 16.25 mL per cell, or 58.5 - 97.5 mL per module (each module has 6 cells).

Doing the Work¶

You'll need to dismantle your battery out of your car to do this. This can be done quickly once you know what you're doing, but follow a YouTube tutorial and take a lot of photos while you do it. Also read the following section and understand what we're dealing with.

Safety¶

This is part in the story where we include the big high voltages can kill warning, but let me add some explanatory detail here: the Prius HV battery is 201.6V nominal - in Australia this is lower then the voltage you use at an electrical outlet every day. But it is a battery - it has no shutoff, and it's DC power (so being shocked will trigger muscle contraction that will prevent you letting go).

Before you do anything to get the battery out of the car, make sure you pull the high voltage service plug, and then take a multimeter and always verify anything you're about to touch is showing 0V between the battery and car chassis.

Now the tempering factor to this is, handled properly, this battery is quite safe to work with once disassembled. High voltage is only present between the end terminals when the bus bars are connected - broken down into the individual modules the highest voltage is 9V from the individual NiMH modules.

Specific Advice¶

What does the High Voltage disconnector do?¶

The big orange plug you pull out of the battery does two things: it breaks the the circuit between positive and negative inside the battery, which makes the voltage at the battery terminals in the car go to 0V. This makes the battery safe to handle with the cover on.

It does this specifically by sitting between the 2 battery modules in block 10, and breaking the connection there. Because the battery output is wired from the last module positive, to the first module negative, this breaks the circuit.

There's a secondary benefit to this once the battery is open: breaking the battery wire here limits the total possible voltage inside the battery to ~130V (from block 1 to block 10). This is still a lethal voltage though.

My Gen 2/2004 Prius HV battery on the bench. Note modules 19-20 don't have a busbar, and are instead disconnected when the service plug is pulled.

The service plug breaks the connection where it does because it has the most benefit for knocking the pack voltages down to more reasonable values. It also disconnects the high voltage at the main battery terminals when this bus bar is broken, because the path between ground and high voltage is broken on the other side of the battery (the terminals connect on the side opposite us).

Getting it down to a safe voltage¶

Until the bus bars are removed completely, the voltage the battery presents remains potentially high. The bus bars make direct contact to the batteries even without the fastening nuts, so until you have removed one of the orange busbars completely, the battery may still have lethal voltages present.

This is a point to stress the issue: unlike your house, there are no circuit breakers or GFCI interrupters here. If you get shocked, it will just keep shocking you and nothing will stop it. Wear lines man gloves, don't touch metal if you're not 100% sure its , when in doubt use a multimeter to check what you're going to ground with tools.

The way to bring the battery down to a safe state is to start the side of the battery with the disconnector switch and remove the bus bars on that side (see image above). They're not connected to wires or other areas so they come off more easily.

Wearing insulated gloves, insulated boots, and using an insulated tool (I used a battery powered drill driver with a hex bit on a low torque settings), knock off all the bus bar nuts working right to left - then, pull the orange busbar off in one go (again, moving right to left, but you are wearing insulated gloves right?)

Important: So long as you only touch a bus bar terminal inside an orange plastic carrier, then there is no high voltage. The greater the distance between any two bus bars though, and the more voltage there is from the batteries stacking up.

After you've removed the bus bars on this side, the battery is now safe - the maximum voltage present is 7.2V.

Important: the battery is still live: you will see sparks (as I did) if you accidentally bridge any two adjacent batteries since you'll be short-circuiting a 14.4V battery (a block). Bus bar nuts are just wide enough to do this.

Be aware that when you re-assemble the battery, past this step (in reverse), the battery very rapidly goes from "safe" to "potentially lethal".

Safety End Notes¶

It's really hard to stress how weird the "safe"/"unsafe" combination of this task actually is. Once the circuits are isolated, this is a safe task. Handled properly, the HV can be quite safely disconnected, many have done it just fine. But ground yourself against that unfused, constantly live 200+ V source, and you are at extremely danger of dying very suddenly. Batteries are not your house electrical: you could probably get away with grabbing live and neutral in a house today, and GFCI would save you. None of that exists in the battery.

Cycling the Batteries¶

Despite the complexity of the actual processes, what needs to be done to the batteries is pretty simple. For this role I used four (4) Tenergy T-180 hobby chargers. These are "okay" as in I was successful with them, though the software implementation leaves a lot to be desired - the hardware seems solid though. They worked for me - and they can discharge (the slowest part) faster then other models on the market. There's 28 cells to do, so optimizing time is important.

Charging Settings¶

As noted the T-180 software isn't great, so your charging settings need to be done in a specific order.

First: goto NiMH mode, then CHG, and scroll down to delta V: you want to this value to 25 mV/C. Notionally this value is in millivolts per cell, but based on my experience I could find no way to set the number of cells, and it didn't seem to be inferred - setting it to a lower value resulted in the charger not properly cycling the cells. Setting the value here changes the value used when using CYCLE mode.

Note: the capacity setting does nothing but try and set the rate of charge. For NiMH the rate of charge is considered to be 1C for a full charge, so about 6.5 amps for the Prius cells, though it has been reported 5 amps might be the better option. I set it 7250mA, then reduced the rate of charge to 6.5A, just in case.

Under CYCLE, you'll want to set the discharge rate to maximum (5 amps) and the minimum discharge voltage to 6V which takes the modules down to 1V per cell, empty. I didn't go lower - if you reverse polarize a cell, that pack is dead and not coming back.

A setting I found important after I started was the wait time setting between cycles. I got much better cycle over cycle capacity gains setting this to 30 minutes then 5 or 10 minutes, and in particular since you are charging to full the modules do get hot. There is also some mechanics of allowing the cell chemistry more time to equilibrate.

Note: When using the CYCLE option, always press START from the second page of settings - otherwise you will not be abe to review and record individual cycle data.

Just the Settings¶

Charge Rate: 6.5A
Discharge Rate: 5A
Discharge End Volts: 6.0V
Delta V Peak: 25mV
Standby Time: 30 mins

Charging Setup¶

IMPORTANT: Never charge cells without compressing them. Prius batteries are prismatic and must have mechanical counter-pressure to prevent swelling. If they swell, the plates will short and warp, and the module will be destroyed. The total side-wall loading (across the whole area) is about 1300 kgs, but basically they mustn't be able to flex what they're in contact with. Keep them in the battery carrier and tightened down or you are very likely to ruin them.

The hardware configuration of a charging setup is important. The T-180s come with many connectors but not simple ring terminals, which is what I wased by adding some crimp adapters. This setup was not ideal: what is under-specified in other literature on the topic is that 6.5A at 9V is quite a lot of current to be throwing through a wire. I used a setup made up of 1.5mm$^2$ house electrical cable (what I had on hand) - this is rated for about 10A at 240VAC, but what this rating misses is at 240VAC you don't really care about 0.3V of voltage drop.

At a much lower voltage, you really do.

Throughout the charging process the chargers would take the batteries up to over 9V from the perspective of the charger, but the battery terminals themselves would be receiving exactly 9 at the end of the process, or only reach 8.8 or so. Which is within spec for NiMH, but means the reported voltages on the chargers can't be trusted.

This also in practice means my 6V discharge setting was probably closer to 6.3V-6.2V which is much gentler but also going to be less effective at restoring the batteries.

If I was doing this again, I would use a very heavy gauge wire (4mm$^2$ or so) and keep the run to 50cm rather then meter or so I used - and use the power supply mode options on the chargers to check voltage drop. I suspect people who have reported very deeply discharging cells may have had it improve their results by this method too - though again, NiMH is unique in that individual cells can safely go to 0V and will come back, but in a series pack you just can't get them all their safely (and if they reverse polarize, you're done).

My charging setup with four Tenergy T-180 chargers. They're angled so their fans don't blow on the adjacent unit.

Two other important things to note:

1. Never charge modules side-by-side. I interleaved modules as I was charging to allow the module under control to use the ones next to it as a heatsink. The one time I charged 3 in a row, they definitely got way too hot.

2. You really need to consider having multiple chargers for this. I had (4) and it took about 2 weeks, though that's including learning time. But generally speaking once you have your baseline, underperforming batteries benefit from more cycles - I ran some modules upto 15 times and they recovered enormously from a position of looking like underperformers, but 5 cycles takes 14+ hours - and there's 28 modules total. If you're going to do this, get multiple chargers - at least 4, and discharge capacity matters - you spend most of your time discharging. Charging takes about 80-90 minutes at the rates I used, whereas discharge takes 5 hours at about 17W average. This is why I went with the T-180s.

Number of Cycles and How to Cycle¶

For most of the work I used Discharge -> Charge cycles. For your first cycle though you want to start out by charging the module with the CHG mode. There's a reason for this: if you have a weak cell or an out of balance cell in your module, the only way to bring it up is to top-balance it by slightly overcharging the pack, which happens when you do a full charge (the Prius doesn't, because overall this reduces cell longevity). If you just use Discharge -> Charge first, then there's a risk that weak cell might get pulled into reverse polarity and just die on you - when it otherwise might have been able to be saved. So do 1 cycle of CHG first.

I did not do this - but I should have. It worked out okay, but the difference between a weak cell and a reverse polarity cell is big. One might restore - one regardless of how good it was is gone, along with it's module.

The other benefit is it makes your data collection easier - you can assume your cells start out with full charge on them for the initial discharge.

Discharge -> Charge cycles have the benefit of ending the module charged, so if you're done with it the last step is to discharge it down to 7.5V to be ready for balancing and reinstallation in the Prius (remember the Prius ECU expects to the cells to be about in this range, and when I removed them this was about the range).

Due to the voltage drop issue, I ended up reinstalling the cells at 7.8V - this worked out fine, as when I pulled them out initially they were charged to about 7.85V from the car.

Step-by-Step¶

What you want to do is get a sense of your battery pack's status quickly:

To start out: Number your cells. You'll be tracking them in a spreadsheet by these numbers. You also have two other fields: Cycles - for individual cycles of each run, and Campaign - the word I used to describe everytime I started a charger cycling a cell. Together I used these to keep the cells in order. The other variables you'll be tracking come from the charger - Charge Vp (Voltage Peak), Charge mAh, Discharge Va (voltage average), Discharge mAh.

Note: While I say number your cells, I actually gave them letters. Since blocks are numbered, I wanted to make modules clearly defined. The schemes goes from A-Z plus alpha and beta. Replacement cells are referred to as the letter plus number - i.e. K1 is K's replacement.

I used LibreOffice Calc and saved my data as tab-separated values (TSV) for ease of use with numpy and matplotlib for graphing and data analysis - which I did in Jupyter Lab (VSCode's Jupyter plugin was excellent for this).

Sample of my data loaded with pandas. You can omit "type" if you're not discharging partially charged cells, which in retrospect I don't recommend. "order" is just my note that it was a discharge/charge cycle.

To start out, get a voltmeter and measure the voltage of all your cells. They should all agree with each other, and any that are about 1.5V out are probably dead from a reverse polarity cell. I had no luck recovering these - so you can save yourself some time by just replacing them right away.

The initial sweep of cell voltages. K is obviously in polarity reversal, and triggered the initial warning light in the Prius.

Experience of Capacity Change While Cycling¶

When I started out I spent a bunch of time worrying I was killing the cells I was working with because the chargers weren't max current delivery limited. This is a real concern, but I was reacting to observations of discharge behavior - and I did not have the delta V cut off set to the 25mV I ended up using. In addition, the voltage drop between charger and battery tended to make the finishing voltages look higher then they were - in my experience the T-180s delta-v detection seemed to work just fine.

Final plots of the data for all cells. Charge/Discharge cycles are corrected so each discharge data point corresponds to the charge datapoint that supplied it's current. There were potentially days between subsequent cycles so self-discharge is probably present.

The cells in my pack overall are in very good shape: I suspect I could've got most of them up to 6000+ mAh, but ultimately I wanted my car drivable again and was content with the results. The big news in this data is that initially under-performing cells showed massive improvements on being repeatedly cycled and this is consistent with other Priuschat members experience.

From this data I am content to say that cycling the modules definitely improved capacity: while the settings are not totally consistent across all cycles, the ones which ended up best started out looking like serious under-performers of the pack. I am actually now 4 modules in surplus since they were looking so poor I thought it would be better just to replace them and save the trouble - instead they improved quicker then my replacements arrived, and I kept them in.

One crucial pattern which stands out is the stair-step discharge performance on the cells which were improving: I saw this a lot, and it was the source of my initial concerns that I was just damaging them. I have no explanation for this from the scientific literature, although my guess would be that since there is an electrochemical potential associated with the gamma->alpha/beta nickel oxy-hydroxide conversion, that once the crystals get small enough they become much more reactive for that cycle of charge.

Final Data¶

The figure below was plotted simply slicing the last set of data I got from the modules, and was used as a benchmark (the red line) for which modules to target for additional cycles).

Final cycle plots. Note K vs K1 which was the replaced module.

Finishing up and Reinstalling¶

Balancing¶

Once I had my cells at a fairly consistent capacity I was happy with, it was time to finish up. This involves balancing the cells - specifically we want to take them all to a similar voltage so the Prius can manage them. The recommendation is to take them 7.5V - with NiMH this is a very weird process though.

From the outline of NiMH chemistry, it can be seen that the voltage of the cells is "stiff" - they don't vary much at all for a very long flat part of their discharge curve, which in turn means assessing state of charge from voltage is tricky.

One suggestion I did see was to simply take them to 7.5V + a known quantity of mAh to get them to a consistent state of charge. The Tenergy T-180s can't do this - they count charge, but can't trigger actions off it. The only control system you have is discharge end voltage.

My solution was to start with my fully charged cells and bring them down - I was aiming for 7.5V (this was the setting on the charger) but wound up with 7.88V or so - close enough to my Prius's charge state when I pulled the battery that I figured it should be okay and the ECU would figure it out.

The other thing I tried to do was balance the cells - this is where you connect them all together positive to postive, negative to negative, in order to let them equilibrate on a common voltage. This process is questionable with NiMH for equalizing charge due to the fairly uniform voltage at multiple possible states of charge.

I made one big mistake here which I realized and undid before anything bad happened: don't do this -

Don't do this to balance cells. It's a bomb.

You can see the problem: I've wired the almost 1.3kWh of battery capacity in parallel, but the common bus bars are haphazardly held maybe 1cm away. It was tricky to wire, it was even worse to unwire since at no state is it not at risk of negative and positive touching and then 1.3kWh of low voltage is going to drop 100 amps into the terminal, or the wire or something. After I did this, I realized I didn't feel comfortable with it, and very carefully unwired it.

Don't do this. It's basically a bomb.

For the time this was connected, I setup one of my Tenergy T-180s to pull 1 amp off the common bus with a target discharge voltage of 7.5V. The idea was that between the current coming off and the batteries, the draw would give the cells a chance to equalize their voltages as the high voltage cells would feed the draw and the pack as a whole would be gently brought towards a common voltage. I only ran it for about an hour before coming to my senses and unhooking it, and after the pack voltage was a flat 7.8V which basically seemed good enough.

There is a correct way to do this: when you open up the pack, flip all the modules around so positive and negative are side by side. No risk of shorts, and you can easily busbar the entire pack with bare bit of copper wire. To implement the constant draw I would crimp a fuse onto the wire and hook it up to the charger.

My conclusion here is that I don't really have enough data. I think my approach to balancing is probably sensible in abstract: by pulling a draw on the pack overall, you can center the cells on a target voltage more quickly then just waiting for quiescient current between them to catch up. But others would say not to even bother once you have matching voltages - which I pretty much did with the chargers in the inital discharge to a consistent value.

Nickel Plated Bus Bars¶

When I pulled my pack out and opened it up, this was the state of the bus bars (after 17 years):

There are a number of Chinese sellers (and American importers) selling replacement busbars which are nickel plated for corrosion resistance. You can just clean up the copper in some vinegar, but since I'm going to the effort and it was about AUD$50 to buy a set of replacements, I ordered some nickel plated bus bars.

Bus bars as they were

Removing the old bars

Installing the new ones

Thoroughly optional, but probably worth it if you're going to this trouble - especially since ensuring nice clean contact areas on your battery reassembly will make things easier.

Make sure you re-install the service plug properly¶

I got the battery back in the car, disconnected the 12V battery for a few minutes to reset the various ECUs - I'm lead to believe this means the car drops its memory of the expected state of charge and will relearn it.

Taking the car out for a drive, firstly it was working properly - the dead cell removed meant it trusted the battery again. It felt snappy and responsive. But, I still had a red triangle on the dash.

Using an OBD dongle and the Dr Prius Android app, when I ran "read battery codes" I found I was getting P3140. This code is not well explained and lots of owners have had it for various hard to tell reasons. In short what it means is "open circuit on safety interlock".

What that means is that there's a sense line somewhere in the high voltage system which is not closed from when the car was started. What sense line? It starts in the high voltage service connector, and runs all the way to the inverter and can be triggered by any problems with that wire loop.

If it breaks while the car is running (say if you crash) then the system will immediately disconnect the high voltage as a safety feature.

But: if it is not connected when the car starts, you just the fault code - and it's a weird code, because most OBD scanners won't see it (FIXD didn't), but Dr Prius can.

Here's the important part: part of this circuit is in the service plug for obvious reasons, but it is only engaged when the service plug is fully installed. The service plug has 3 actions to install it: insert, lift up to engage, and then pull down to lock. The final step - down to lock - engages the pins on the sense line. When I initially put the service plug back in, I didn't do the last step - the car would start, and run, but I would get the problem light and it would not be cleared.

Service plug installation procedure

tl;dr: If you've pulled the service plug for any reason, and have a warning light that won't clear - check you have fully engaged the service plug.

Conclusions¶

My car is working again. I'm at least $1500 ahead after parts and chargers, probably more accounting for labor, displacement etc. and frankly this is just the sort of thing I like doing.

I've replaced two modules which went into reverse polarity on a cell in the pack. I am wondering if an earlier recondition might have saved them.

The car is only just back together, so we'll see how the situation evolves.

Why?¶

The key to me for blogging is to keep it simple. This at first meant static sites that rendered nicely, and this was a function fulfilled by Wintersmith.

The Problems¶

The problem with Wintersmith was that it was a pure markdown solution. At the end of the day, it turned out that most of what I wanted to talk about probably had some type of interactive or graphical component to it, or I just wanted to be able to add photos and images easily.

None of this is easy with pure markdown.

The Solution¶

The new solution for me here is Nikola which seems to hit a new sweetspot of codeability versus low friction. Specifcially it's support for posts being rendered out of jupter-lab notebooks, something that I've had just grow in usefulness.

Perceived Benefits¶

Getting the transition from "jupyter notebook" to "blog" post down to as frictionless as possible feels important. When you want to share neat stuff about code, it helps to do it right next to the neat code you just wrote.

When you want to share data analysis - well that's a jupyter specialty. And when you want to show off something you're doing in your workshop, then at the very least pulling in some images with jupyter is a bit more practical and probably a lot more likely to be low enough effort that I'll do it - this last part is the bit that's definitely up in the air here.

Workarounds¶

Keeping media and posts together¶

The first problem I ran into with Nikola is that it and I disagree on how to structure posts.

Namely: out of the box Nikola has the notion of separate /images /files and /posts (or /pages) directories for determining content.

I don't like this, and it has a practical downside: when working on a Jupyter notebook for a post, suppose it requires data or I'm using some tool which will do drag and drop images for me? What I'd like is to open that notebook, and just that notebook in Jupyter or VSCode and work on just that file - including how I reference data.

Although this issue suggests a workaround, it has a of drawbacks. But more importantly - one of the reasons we use Nikola is it's written in Python, and better - it's main configuration file conf.py is itself, runnable Python.

This gives us a much better solution: we can just generate our post finding logic at config time.

To do this we need to find this stanza:

POSTS = (
    ("posts/*.ipynb", "posts", "post.tmpl"),
    ("posts/*.rst", "posts", "post.tmpl"),
    ("posts/*.md", "posts", "post.tmpl"),
    ("posts/*.txt", "posts", "post.tmpl"),
    ("posts/*.html", "posts", "post.tmpl"),
)

This is a pretty standard stanza which configures how posts are located. Importantly: the wildcard here isn't a regular glob, and all these paths act recursive searches, with the directory names winding up in our paths (i.e. posts/some path with spaces/mypost.ipynb winds up as https://yoursite/posts/some path with spaces/mypost-name-from-metadata)

So what do we want to have happen?

Ideally we want something like this to work:

|-my-post/- my-post.ipynb
         |- images/some-image.jpg
         |- files/data_for_the_folder.tsv 

and then on the output it should end up in a sensible location.

We can do this by calculating all these paths at compile time in the config file, to workaround the default behavior.

So for our POSTS element we use this dynamic bit of Python code:

# Calculate POSTS so they must follow the convention <post-name>/<post-name>.<supported extension>
_post_exts = (
    "ipynb",
    "rst",
    "md",
    "txt",
    "html",
)
_posts = []
_root_dir = os.path.join(os.path.dirname(__file__),"posts")
for e in os.listdir(_root_dir):
    fpath = os.path.join(_root_dir,e)
    if not os.path.isdir(fpath):
        continue
    _postmatchers = [ ".".join((os.path.join("posts",e,e),ext)) for ext in _post_exts ]
    _posts.extend([ (p, "posts", "post.tmpl") for p in _postmatchers ])

POSTS = tuple(_posts)

PAGES = (
    ("pages/*.ipynb", "posts", "page.tmpl"),
    ("pages/*.rst", "pages", "page.tmpl"),
    ("pages/*.md", "pages", "page.tmpl"),
    ("pages/*.txt", "pages", "page.tmpl"),
    ("pages/*.html", "pages", "page.tmpl"),
)

Testing this - it works. It means we can keep our actual post bodies, and any supporting files, nicely organized.

Now there's two additional problems: images and files. We'd like to handle images specially because Nikola will do automatic thumbnailing and resizing for us in our posts. They're handled lossily. Whereas files are not touched at all in the final output.

The solution I settled on is just to move these paths to under images and files adjacent to the posts respectfully. This means is the Jupyter notebook I'm using is referencing data, it's reasonably well behaved.

For files we use this config stanza:

FILES_FOLDERS = {'files': 'files'}

for e in os.listdir(_root_dir):
    fpath = os.path.join(_root_dir,e)
    if not os.path.isdir(fpath):
        continue
    FILES_FOLDERS[os.path.join(fpath,"files")] = os.path.join("posts",e,"files")

and for images we use this:

IMAGE_FOLDERS = {
    "images": "images",
}

for e in os.listdir(_root_dir):
    fpath = os.path.join(_root_dir,e)
    if not os.path.isdir(fpath):
        continue
    IMAGE_FOLDERS[os.path.join(fpath,"images")] = os.path.join("posts",e,"images")

Setting up the publish workflow¶

Nikola comes out of the box with a publishing workflow for Github pages, which is where I host this blog.

Since I've switched over to running decentralized with my git repos stored in syncthing, I wanted to ensure I only pushed the content of this blog and kept the regular repo on my local systems since it leads to an easier drafting experience.

I configure the github publish workflow like so in conf.py:

GITHUB_SOURCE_BRANCH = "src"
GITHUB_DEPLOY_BRANCH = "master"

# The name of the remote where you wish to push to, using github_deploy.
GITHUB_REMOTE_NAME = "publish"

# Whether or not github_deploy should commit to the source branch automatically
# before deploying.
GITHUB_COMMIT_SOURCE = False

and then add my Github repo as the remote named publish as

git remote add publish https://github.com/wrouesnel/wrouesnel.github.io.git

and then synchronize my old blog so nikola can take it over:

git fetch publish
git checkout master
find -depth -delete
git commit -m "Transition to Nikola"
git checkout main

and then finally just do the deploy:

nikola github_deploy

Next Steps¶

This isn't perfect, but it's a static site and it looks okay and that's good enough.

I've got a few things I want to fix:

• presentation of jupyter notebooks - I'd like it to look seamless to "writing things in Markdown"
• a tag line under the blog title - the old site had it, the new one should have it.
• using nikola new_post with this system probably doesn't put the new file anywhere sensible - it would be nice if it did
• figure out how I want to use galleries