Friday, 6 September 2013

Mohi Mete: Is The Internet Down?

Karyn’s Introduction:

Our journey over three years has had a full gamut of both joys and tribulations. 15 weeks to prepare a school for opening alongside appointing all the staff is no easy feat and we did ‘build a lot of the plane while flying it.’ Nowhere was this probably truer than IT infrastructure and systems.

We managed to employ a real range of teaching staff from throughout New Zealand to join us on this journey. 
Some of them were enticed by the thought of a new school- little did they know we would still be waiting for that three years on! 
Some were enticed by the thought of living and teaching in a small rural community. 
Some were enticed by the vision that we had for developing real future focussed learning.

Whatever their motivation in applying for a teaching position with TKAS our staff have worked tirelessly over the last three years to build a school with our learners together. They are an amazing group of educators, and it is to their credit that through the trials and tribulations we started Term 3 2013 with 14 of the 17 establishment teaching staff still with us.  There is an amazing staff unity that has developed over these years, and a great willingness to provoke and prompt each other to ‘unlearn’ the more traditional ways of teaching and learning and to experiment and try new ways of teaching and learning.

However often those staff have found themselves in roles they weren't originally employed for through pure necessity.

Mohi, is one of the original 17 teaching staff members and while we originally employed him to be a teacher, and he still is part of the week, we were absolutely rapt when he proved to have such a handy background in IT and as an electrician and his skills have been vastly called upon.

This is Part 1 of our eLearning story as told by Mohi.

Part 1 of 3 - The First Year (2011)
No Blog about the journey that we have been on over the last few years at Te Karaka Area School would be complete without some record of the trials, frustrations and lessons that we have learnt about ICT systems and infrastructure. And given that I have had the most to do with those systems, I guess I had better tell the story.

In The Beginning
The job application pack contained a lot of info, and gave an outline of creating a modern eLearning School, which I found hugely attractive, it also had an IT skills questionnaire, and being a geek I naturally answered every question as "highly proficient", not realising how much that proficiency would eventually be called on.
It really began though before most of us were appointed, as 'we' inherited some gear and systems from the previous schools, Te Karaka Primary School and Waikohu College. The most obvious to start with was our Student Management System, "Kamar" which us Junior School teachers were told was the only thing that the Secondary Teachers would use. Like it, or lump it!

A fair bit of work had happened before we arrived, because on Day One we had a wireless network, network file servers, a network printer/photocopier, Internet access, email through Google Apps for Education, LMS with Ultranetand we were expecting the arrival of a batch of new HP and Macbook laptops. The only thing we didn't have was our teacher laptops as we had all just been appointed. Unbeknown to us at the time, we wouldn't have them until April, which meant we basically went the whole first term without them. (paper rolls and runners to the office every period.....yay!) The students fared somewhat better, with the new HP and Mac laptops arriving in week 4 and being distributed into classes by week 6 along with some older Notebooks and Netbooks that we had inherited from the College. Students were able to log in and get onto Mathletics, browse the web for research, make Office documents and with the Macs even do some editing in iMovie. A personal highlight being our Junior School end of term Assembly where groups presented short film clips that they had been working on. The total engrossment of all the students as they watched themselves in movies that they had recorded and then edited was an incredible buzz.

Triple Rain Fade
The second term was a step up, we had our Teacher laptops and could actually do our roll straight into Kamar and even receive and send notes and notices through Kamar. The students were using online tools and various applications and things seemed to be going along pretty well.
That was until it started to rain. Yes Rain! Never heard the term Triple Rain Fade? Let me explain.
Our internet connection was a Wireless link that came in at the old College site and was then relayed to the primary by another wireless link, and on a good day it was capable of download speeds of around 7-8Mb/s (Mega bits per second).
But when it rained it dropped to less than half. So that was Rain fade x 2. Once for the incoming link and the twice for the link between the schools.
Our wireless Access Points that covered the school were 4 Mikrotik units in weatherproof boxes mounted outside the classrooms. Being outside they were also affected by the rain. Rain fade tripled! Which pretty much meant that when it rained, network, and internet performance went down to dialup speeds. 
"Mr Mete my thing is stuck!" Sigh, "Yes, just wait for the rain to stop"
That year it rained, …… a lot!
Along with that, someone decided they needed our laptops more than we did and broke into the school and took several of them. Twice!!
Hard Lesson Learnt:
1. If you have lockable laptop carts, make sure that you put padlocks on them to keep the laptops in the carts.
While this may seem like a pretty straight forward thing in most schools, the reality was we all had a thousand things we were trying to get sorted and this just hadn't been sorted yet.

Where are our wheels, bro?
To get around the problem of the local wireless rain fade in my class I found an old 24 port Ethernet switch that was in the old College and pressed that into service, with students having to plug in via patch leads to get a reasonable connection. We also got some of the old HP desktops we inherited from the College and pressed those into service in the Junior Spaces. Because they were hardwired they weren't affected by the wireless rain fade. The problem was partially solved but everyone was still badly affected by the combined rain fade issue. (roll on Spring and finer weather)

As we progressed through the year and tried to implement more and more digital and tech related learning/teaching initiatives we kept uncovering more and more problems that needed sorting or some sort of ad-hoc arrangement to make things usable/work. 
The Macs laptops were a major pain to update as we didn't have a Mac server to cache the updates, so in came my iMac from home to become our server.
Getting 40 iPads may have been a bit more adventurous than we originally thought, but somehow we managed to get things sorted with the iMac serving as the main synch station, and pods of iPads going out into classes to try out new ways of getting kids engaged and learning.
The Windows install on the laptops was so locked down that no student or teacher could plug their laptop into a projector to show anything because the Display control panel was not accessible. The Windows installation didn't load the correct video drivers on the laptops so the desktops were skewed, it also didn't load a sound driver on the Netbooks so they had no sound. Then if I installed them individually on each device, which was a very time consuming process, the next time Windows update ran, ....... it removed them!!!

Then just to make life fun, every now and then half the network would go down and we would have to climb up to a cupboard in Room 3 to reset a little network switch. (or plug the lead back in if one of the cleaners had unplugged it to do the vacuuming)
Being the unofficial 'Tech Guy' also made things interesting as students and teachers figured out who was the best person to go to whenever they had an issue. It became enough of a problem that Karyn sent out an email asking for teachers not to bug me, or send students to me during class time unless it was ultra urgent. Not that I minded helping people, we all had things we wanted to achieve, and my class certainly learnt how to get on with things if I was busy helping someone. (well most of the time they did)

By midway through Term 3 however, it was 'feeling' as though the tyres had not just gone flat on our 'digital school bus', but that we had no tyres at all, the motor had overheated and we were pulling things along on the rims with several bits of baling twine tied together! That's not to say things didn't actually work, they did. However the effort required to keep the system going and semi-usable was not sustainable longer term. With the start date for the refurbishment of the College actually becoming a question of whether to refurbish at all, or to do a total rebuild, the reality was that we were going to be on the Primary School site for a lot longer than the 12 months we had originally thought we were going to be there. Oh Joy!

What 'exactly' do we have?
About that time, Karyn came to me to discuss what we could do, bearing in mind that we didn't want to spend lots of money on infrastructure where we were because we would just have to leave it behind when we did move to the new school. 
The main problem as I could see it, was that we didn't really have any idea or overview of what our IT infrastructure looked like and neither did anyone else. 
Our Tech support was in Auckland and had only been here for a day to set the server up and didn't want to know about cabling or wireless problems. The guys who did the wireless were just contractors to our ISP and had no real idea of the cabling set-up, and our ISP would only help with wireless, didn't want to know about server issues, which left us stuck in the middle getting frustrated because it didn't work the way we needed it to. So the upshot of the discussion was that I would do some investigating and work out what we actually had, and formulate a plan for what to do going forward. So that is what I did, whenever I could find time.

It took me until late in Term 4, and culminated in a 3 page report on the state of our IT infrastructure, systems and support. Karyn and I used that for a long discussion on how best to proceed. The biggest problem was the cabling in the school. The original installation was never anything amazing, and it had just been added to in haphazard methods by various different contractors over the years, with the result that instead of a flat star type network we had a long daisy chain of devices strung together with multiple points of failure able to bring our network crashing down.


So Priority One was to fix the cabling. Luckily, I am an ex-electrician, so I knew how to deal with that, but it would have to wait until after School finished.
The second major issue was a combination of who had control and software platform.
The network software platform was by an English company, which according to their sales guff was designed to make life managing a network simple for Teachers. Yeah Right! 
It might well have been if it was set up properly, but our set-up just didn't work, and trust me when I say I spent a lot of time trying to make it work.
The other aspect was the remote "support". When I sat down and actually analysed the support contract that we were paying a monthly fee for, it boiled down to we were basically paying for email support. Which was rather expensive and explained why nothing actually got fixed.
The last straw was when I actually got the admin password for the Server and logged in to check the setup of the backup software.
On looking through the logs I was horrified to find that only one of the backup jobs for the last 12 months was fully successful, and in fact it was the original one that had been done when the system was first installed!
Lessons Learnt:
1. If you have a Support Contract, demand some reports of what maintenance is carried out each month. (don't accept generic descriptions of work!)
2. Consolidate your support so that responsibilities are clear and get you results, quickly.
3. Make sure you have all the passwords needed to take control if necessary, on site.

So the plan was to ditch the Support Contract, get rid of the existing network software platform and replace it with a standard Microsoft Server setup running on VMware and vanilla Win7 installs on laptops and desktops. To get control of the wireless access points I would just reset them to factory defaults because the password the guys that set them up gave me didn't work. In order to be able to do all this it was agreed that I would need to come out of the classroom and go into a full ICT Administrator role. Not exactly what I signed up for, however, we had a vision for the way we wanted the school to be and I was determined to do whatever I could to make that a reality.

Merry xMas
We went ahead and cancelled our 'Tech Support' contract, I had taken control of the wireless access points, and we had organised for the Incoming Internet link to be shifted to the Primary Site over the holidays, with the wireless link between the schools to be repositioned so that we could send the network across to Karyn's house as some of the rooms were going to be used as classrooms. I started getting ready for all the changes by downloading tons of software, studying up on VMware, Windows Server 2008 R2, Active Directory, Group Policy, Kamar, Key management servers, Symantec Anti Virus, roaming profiles and a bunch of other stuff that would go into the new setup. I also made sure we had complete back-ups of all our files and a full back-up of the existing set-up in Veeam on our NAS.

The last day of School arrived which was only a half day for the students. We had a nice little final assembly and sent the students off home.
The teachers all gathered in Room 1 for a shared lunch, some well deserved drinks, and a last debrief for the year. Then just as we were all starting to relax, ping, the power went off. Which wasn't a big deal as it had happened a few times throughout the year. However after 25 minutes with no power I thought I better go and shut the server down before the UPS ran out of battery power. I had just logged in when, klunk, the UPS died and the server crashed, Hard! Oops! (and I thought "Man I'm glad I did that backup")
Amazingly, we had made it through the whole year without that happening, until the very last day! Karma? You decide!


  1. Wow! :D You do a massive job brother Mohi! Keep going hard as we know you will...under that rough exterior lies a pretty nice guy! :D
    Thanks for sharing!

  2. "For unto whomsoever much is given, of him shall be much required...."

    That will teach you to be so talented, competent and capable.

    Now if only you had a friendly welcoming smile....