SIP Closures

After I posted an earlier version of this script I realized it didn't display closures.  It was biased because I was already looking for certain bad actors.  In an attempt to broaden a bit this version includes information about mux_closures the server is performing among the ten busiest IPs.

#!/bin/bash
# may not play nice with really huge logs unless your station has plenty of RAM

printf "\n\ntop 10 ips with inbound messages\n"
printf "=================================================================\n"
grep "inbound message" $1| grep -Eo "([0-9]{1,3}\.){3}[0-9]{1,3}" | sort -rn | uniq -c | sort -rn | head

 
printf "\n\nlast line of the top 10 inbound ips\n"
printf "=================================================================\n"
for i in `grep "inbound message" $1 | grep -Eo "([0-9]{1,3}\.){3}[0-9]{1,3}" | sort -rn | uniq -c | sort -rn | head `
do
        # echo 'i = '${i}
         if [[ ${i} =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then
                tac $1 | grep "inbound message" $1 | grep -m1 ${i}
    fi
done


 
printf "\n\ntop 10 account logins\n"
printf "=================================================================\n"
sed 's/^.*sip//' $1 | grep "Successful login" | sort -rn | uniq -c | sort -rn | head

 

printf "\n\ntop 10 mux_close IP addresses \n"
printf "=================================================================\n"
# grep "mux_close" $1 | sed 's/^.*::ffff://' | grep -Eo "([0-9]{1,3}\.){3}[0-9]{1,3}" | sort -rn | uniq -c | sort -rn | head


declare -a ips
# IPs of mux closures
declare -a muxes
# number of entires of those IPs in the logs

mux_lines=($(grep "mux_close" $1 | sed 's/^.*::ffff://' | grep -Eo "([0-9]{1,3}\.){3}[0-9]{1,3}" | sort -rn | uniq -c | sort -rn | head ))

length=${#mux_lines[@]}

declare -i tx=0

for ((t = 0; t != length; t=t+2)); do    
        muxes[tx]=${mux_lines[t]}
        ips[tx]=${mux_lines[t+1]}
#        echo ${muxes[tx]} " " ${ips[tx]} " " $tx
    tx=$tx+1
done


# values of the first mux_close of each of the top ten
declare -a firsts 
declare -a first_lines
tx=0
for ((t = 0; t != length; t=t+2)); do
    first_lines=($(grep "mux_close" $1 | grep ${ips[tx]} | head -1 | sed 's/^.*; //' | sed 's/\ remain.*$//'))
        firsts[$tx]=${first_lines[0]}
    tx=tx+1
done

# values of the last mux_close of each of the top ten
declare -a lasts
declare -a last_lines
tx=0
for ((t = 0; t != length; t=t+2)); do
        last_lines=($(grep "mux_close" $1 | grep ${ips[tx]} | tail -1 | sed 's/^.*; //' | sed 's/\ remain.*$//'))
        lasts[$tx]=${last_lines[0]}
        tx=tx+1
done

 

# print output
tx=0
for ((t = 0; t != length; t=t+2)); do
        echo -e "mux_close entries" ${muxes[tx]} '\t' "IP Address " ${ips[tx]} '\t' "first entry " ${firsts[tx]} '\t' "last entry " ${lasts[tx]}   
        tx=tx+1
done

2015 Hack-A-Way Dates

After much deliberation, forecasting, haruspicy the dates for the 2015 in Danvers MA (just north of  Boston) have been selected and it will be November 4th - November 6th.  Many thanks to Kathy Lussier for her tireless work on this!

And please ignore anything she says about her own efforts, she's too modest.

 

SIP Summary Script

I suspect that most folks who have cause to need it have already formed solutions of their own.  But for those who haven't I present a bash script for summarizing the top 10 IPs with inbound messages, those actual entries (to see how many connections are still open) and the top ten accounts logged in by occurrences of login.  It's pretty simple but handy when you're getting hit by a rogue SIP device.

#!/bin/bash
# may not play nice with really huge logs unless your station has plenty of RAM
 
printf "\n\ntop 10 ips with inbound messages\n"
printf "=================================================================\n"
grep "inbound message" $1| grep -Eo "([0-9]{1,3}\.){3}[0-9]{1,3}" | sort -rn | uniq -c | sort -rn | head
 
printf "\n\nlast line of the top 10 inbound ips\n"
printf "=================================================================\n"
for i in `grep "inbound message" $1 | grep -Eo "([0-9]{1,3}\.){3}[0-9]{1,3}" | sort -rn | uniq -c | sort -rn | head `
do
        # echo 'i = '${i}
         if [[ ${i} =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then
                tac $1 | grep "inbound message" $1 | grep -m1 ${i}
    fi
done
 
printf "\n\ntop 10 account logins\n"
printf "=================================================================\n"
sed 's/^.*sip//' $1 | grep "Successful login" | sort -rn | uniq -c | sort -rn | head

Evergreen Conference 2015

Another annual conference has come and gone.  2015’s seemed short.  It wasn’t, in fact, but the time seemed to go quickly.   I admit I’m not as good a traveler as I once was.  Part is due to age, and part is routine.  I always wake up the same time of day.  Even daylight savings causes me difficulty adjusting.  Flying to a different coast and telling my body to adjust three hours is nearly impossible.  Then five days later I do it in reverse.  In between I’ve run nonstop for days on end.  I interact with people and burn every ounce of introvert energy I have.  I also run on little sleep, staying up late and getting up early.  And it’s worth it.  By the time I drag home I feel a bit like Toshiba MIfune in Yojimbo, crawling exhausted and battered under the house hoping to just get away from everything and recuperate.

Toshiro Mifune from Kurosawa's Yojimbo beaten up.   How I feel after a conference.

Toshiro Mifune from Kurosawa's Yojimbo beaten up.   How I feel after a conference.

I point that out simply because everything is made better by an Akira Kurosawa reference.

But, I do it each year because it’s worth it.  This year I taught a half day SQL workshop, I served on a panel welcoming new folks to the Evergreen community and I did a presentation on extending data sources in the reporter.  You can see all that from the conference schedule.  But it’s far more than that.  What I gain isn’t entries on a vita.  Even a few years ago attendees of my SQL and Reporter workshop would have been staff with very tech oriented roles.  The sessions were full with 20+ people in each and they were librarians.  Yes, tech curious but by no means systems administrators - traditional librarians who want to dig deeper and deeper into the power that Evergreen can provide.  I like to think I helped make some materials more accessible to them and the fact that this new power user class is growing in the community is a wonderful thing.  That additional depth and breadth in the community is a healthy thing.  It means that the idea of a tech curious librarian is increasingly irrelevant.  Every year that I use a phrase like that it sounds sillier and sillier and I’m happy for that.  

Evergreen SQL Workshop.

Evergreen SQL Workshop.

And to paraphrase Billy Shakespeare, the community is the thing.   Attending the reporting interest group I talked about the need for new core reporting features with staff from all over the country (and world), about the need for existing and new libraries.  I think we need to bring back core reports into Evergreen as something that is expanded and tested with each version and it’s something I hope to work on this year, starting with going through the ones that were developed for 1.6 and updating them.  I talked with folks from Indiana about homebound services and something vaguely (but not quite) like plans were made. But talking is a starting point.  I talked about philosophies and their practical import.  

Many bad jokes were made (by me) and a few good ones (not by me).  We compared war stories, planned for the future, discussed what ifs and shared discussions about the meaning of life, or at least governmental ethical obligations and spending regulations.  Talking to other consortiums is always illuminating.  So is playing board games late into the night (I won Stone Age but it was a bit unfair since most of the others hadn’t played it before).

Good folks to hang out with.  Your ILS is in good hands.

Good folks to hang out with.  Your ILS is in good hands.

I left after three years on the Oversight Board having completed a three year tour of duty.  Several folks expressed surprise at my tour ending.  Three years go quickly.  A few also asked why I didn’t run again.  The truth is that we instituted the format of rotating members off the board so that it wouldn’t become stagnant.  Our community is large and diverse.  I want to let new voices in.  I may run again in a year or two.  I may not be able to vote but I’m still around and I promised to come in and sit in on meetings when time allows.  I also agreed to remain on the merchandising committee and to assist the board with some special issues if they come up.  

My last Oversight Board meeting of this tenure as a board member.

My last Oversight Board meeting of this tenure as a board member.

I’m also making some changes to the Hack-A-Way.  Submissions are now open for 2015 and will remain open until June 19th.  However, we are moving to an annual model for the Hack-A-Way.  With it now in it’s fourth year it’s become an institution.  As the kickstarter of it I still think of it as a scrappy little thing that has to prove itself so seeing folks planning far in advance and competing to host it surprises me.  But, it shouldn’t.  I myself have pointed out the good work that has come out of it each year.  So, a year wraps up and I head home to recuperate.  

I had to leave before the developer update was done but I know the gist already.  The new staff client looks amazing.  I would be tempted to say that we should do a second (unusual) upgrade in 2015 but with so many other projects on our plate it’s probably not in our stars.  And maybe it’s best to just go over to the new staff client all at once anyway.  The new infrastructure also opens a lot of new doors I think.  But all that is left behind as I fly back to the east coast and just worry about getting gate to gate.

Today I returned to work, jet lagged and exhuasted.  But in a way the conference lingers, it’s effects reverberate in strange frequencies and conversations will continue on in IRC and by email for weeks and months to come.  Really, we think of the conference as a distinct moment in time but it’s more of a peak of a sine wave that goes on and on.

 

Evergreen Conference 2015

It's Thursday.  Crap.

That means that one week from now I'll be at the Evergreen International Conference on a panel saying "Welcome to the Evergreen Community" in the Mountainview Room.  Ymail Suarez of the Berklee College of Music, Grace Dunbar of Equinox Software and Andrea Buntz Neiman of Kent County Public Library are being kind enough to let me sit with them.  Eight days from now I'll be talking about data sources in the reporter.  Nine days I'll try to sort the blur into my mind as I head back across the country.  

Before all that though, six days from now I'll be doing a workshop on SQL at the pre-conference.  I put out a poll for requesting things that people will be interested in seeing a few weeks ago and here they are sorted by difficulty.  I won't actually decide what to work on until we get there and I have input from the audience but these are possibilities based on those who submitted to the poll.  For good or ill this is going to be done live with only a tiny bit of preparation so attendees will get to participate in real world report writing exercises.  

Warmup 

These are reports that are very straightforward and we will use to highlight syntax and get the synapses firing, which is to say let caffeine kick in.  

  • Change permission group according to patron age (birthdates)
  • Cancel a bad hold
  • Delete copy locations
  • Audit reports -- patron records with missing/null data in specified fields

 

1 Cup of Coffee

These reports are straightforward without difficult transforms or complicated joins but may require you to be a little more awake to notice the (in hindsight sometimes) obvious.

  • "untargeted" holds -- i.e. records/items with holds on them where all copies have gone lost/missing/claimed returned/etc. Ideally run regularly so we can promptly follow up with ILL or replacement.
  • Reports making use of local stat cats & their variables
  • Extract circulation stats for a specific month/year

Will Need Edits

Now we get into the reports that require a bit more planning and we may find ourselves going back and tweaking a fair bit.

  • Purchase alert report *
  • Compare number of active holds on a title to number of holdable items on same title**
  • Identify active holds that have remained unfilled despite newer holds on the same title being filled
  • Weeding report *

* These two reports are one that I have well traveled versions of.  We will probably do these and I'll start with the planning and though process but part way skip and show my completed versions and how they work.  

** This is really a subset of the purchase alert but I'll show how it can be looked at a little differently if the crowd is interested.

Lacking Context

I had one I don't have the context for to understand - Hiding items in a collection.  If the person who wrote that reads this feel free to contact me.  It sounds interesting!

So, a weeks time, far to go and much to do.

 

Hack-A-Way 2014 Wrapup and Photos

I've uploaded my photos from the Hack-A-Way to a gallery on my site and (more importantly) the Evergreen community Flickr account.  See them along with somewhat lame commentary here:

https://www.flickr.com/photos/evergreen-ils/

You'll notice in the photos a lot of people quietly typing.  There was discussion but the nature of a hackfest is a lot of collaboration and coding.  And this hackfest had coding, tutorials, documentation and more.  Indeed, many people came in with things to talk about, things to resolves, things to learn and things to work on together.  It was great.  One person said they wished we could do this several more times a year.  That's probably not practical but the fact that it gave that feeling of being useful made me feel good.

It was a good but exhausting week.  I started with picking up materials the Saturday before and it just went on from there.  This isn’t to say that I did it all.  Other staff, at the York County Library, were critical to pulling this together and although their roles were sometimes invisible to participants, trust me when I say that everyone appreciates each of their efforts immensely.   For me it wrapped up just a few hours ago, a week later, dropping off a few colleagues at the airport and doing this blog entry.  

Several people commented on how productive it was and big progress was made on several fronts. 

http://wiki.evergreen-ils.org/doku.php?id=hack-a-way-2014

The Evergreen wiki page and linked collaborative Google Doc outline a bit of what happened.  I also tried to highlight some of the more offbeat moments on Twitter under the #egils and #hackaway14 hashtags.  Well, at least the PG rated events.  The exact language a few points may not have been copied verbatim.  I think that would have raised it to PG-13 in one or two cases.  And I do regret not getting the beat boxing on video.

We didn’t fix the entire world’s (or even all of Evergreen’s) problems but we made progress.  We looked at Evergreen issues and compared issues with specific installations.  We talked about big picture issues that affect the future of the community.  We groused, we pontificated and just shared opinions.  And we ate BBQ. 

We talk about community in open source a lot but when we talk abstractly it’s about faceless sources of email and git commits.  Events like this, even more than the conferences, bring home how human that community is.  I’m lucky in that I like these humans.  I like spending time with them and like working with them but it still makes for a very long week. 

I learned a lot of new things this year that I hope to put into practice over the next year and soon enough #hackaway15 will start it’s own planning process.

 

 

Hack-A-Way 2014 Day 1

I've been organizing the Hack-A-Way for three years, since it began, but this year it's come to my own library in Rock Hill, SC.  SCLENDS has been active in the Evergreen community as much as our resources could allow from the beginning and this has been the first time we've had an Evergreen community event in South Carolina.  While I've been both happy and proud to host it myself this year it also reminds me of how much effort past hosts (Equinox, Calvin College) put into it.  We've learned each year from it and it's evolved.  

While Hack-A-Way was originally conceived of as a two day event with a "pre" day like a pre-conference I think it's time to simply change that idea to a three day event with the acknowledgment that some people may arrive at various points during the first day.  I've also in the past not started looking for hosts until after the annual conference.  As it's been a fairly low key event with a small group of technical members of the community I didn't see it as needing a lot of lead time.  Of course, the numbers of attendees has grown (though not dramatically) and the standards for hosting have been raised by the first hosts.  Now, I think I will start looking for hosts earlier, maybe as soon as when this one is over.

We did a lot on the first day, alternating between group discussion and working together on small projects.  We attempted to extend our remote participation via jit.si but tomorrow will fall back on Google Hangouts.  Tragically, love for FLOSS projects sometimes has to bow to effectiveness.  And, as usual, we use IRC.  Some of the topics can be found at the Evergreen WIKI at http://wiki.evergreen-ils.org/doku.php?id=hack-a-way-2014 where you can also find the working Google Doc we are taking notes at though more happened not quite captured there.

You can also follow along on twitter using the hashtag #hackaway14

And now, the day, in brief, in pictures,

It turns out that a bunch of developers and Linux admins are the wrong people to troubleshoot Windows.  "Charms bar?!?  Is it really called that?" was said at one point.

It turns out that a bunch of developers and Linux admins are the wrong people to troubleshoot Windows.  "Charms bar?!?  Is it really called that?" was said at one point.

I didn't trust the wireless so I provided a gigabit switch with plenty of cables.

I didn't trust the wireless so I provided a gigabit switch with plenty of cables.

Do you trust the future of your ILS to these guys?

Do you trust the future of your ILS to these guys?

Let's backport that, what could go wrong?

Let's backport that, what could go wrong?



SQL for Librarians

Here it is, SQL for Librarians.  I closed out the Cambridge Evergreen Conference (for good or ill) and actually keep a few folks there until 12.  I had a lot of great comments so I think it was fairly successful despite being a tad loopy from allergy medication.  And I blame the medication for a few things that upon listening to this I cringed at.  In a perfect world I'd love to do this again and do it with a full workshop format.  

Slides: http://www.slideshare.net/roganhamby/sql-for-librarians

Youtube: https://www.youtube.com/watch?v=3Iz-HFiDq6E

Conversations At ALA About ILSes

Normally I leave my rabid pro-FLOSS pro-Evergreen attitude for the web.  In person I make a conscious effort to not be so forward as it's usually a hinderance to meaningful conversations.  Today at ALA in Vegas I threw that rule out the window.

I didn't do it right away but eventually I was worn down.  Worn down by what you ask?  Since this morning, I've had six conversations with people bitching about their ILSes.  And their complaints were legitimate.

"I went to Blue ILS years ago and it was great but the founders left and they're now evil corporate sociopaths who abuse us regularly."

"I was about to go to Red ILS which is great with great support but they just got bought out by evil sociopaths and I don't feel good about this anymore."

Valid concerns.  What annoyed me was the fatalism.  "Whatcha gonna do?"  Go open source.  There, I solved it for you.  I told the last one that in those terms.  I usually say it anyway but with more respect for the difficulties their situations face.  But I'm tired of having those issues used as excuses for why libraries should allow themselves to be abused.  The difficulties make things non-trivial, maybe even hard, but not impossible.  And it is the answer.  If you're not being abused, you just wish it was better and you're willing to live with it because you have higher priorities then that's fine.  But if your voice sounds like you're beaten regularly when you talk about your ILS vendor  ... yeah, you need an intervention.

So, how do you do this?

Well, you could host yourself in which case you only have to trust yourself.  But, that may not be efficient.  I use hosting from Equinox Software.  My hosting and support provider has the advantage of expertise from hosting many installs and economies of scale.  Why do I use Equinox? Because I trust them.  Why do I use Evergreen and open source?  Because I don't have to trust them tomorrow.

Implicit in the complaints is lock in - whoever they go with for support owns the software.  Changing support means changing software which is a huge deal.  But when no one owns it, everything is different.  My contract allows free access to my data.  If the leadership changed at Equinox I would just change service providers.  My users won't know.

And yes, that's why open source is the answer.  There's no reason for anyone to ask me if I'm happy with my ILS support because if I wasn't, I'd just change it, year to year if I had to.  And that's a very good thing for my library.