Thursday, September 17th, 2015 | Author:
  • Part 1 – IntroductionSetting up Simple Queues (This post)
  • Part 2 – Reliably Identifying trafficSetting up Mangle Rules (Coming Soon TM)
  • Part 3 – Priorities and LimitsSetting up Queue Trees (Coming Soon TM)
  • Part 4 – Monitoring UsageRedefining QueuesLimiting Abusive Devices (Coming Soon TM)
  • Part 5 – ??? Profit ???

Introduction

The first problem one usually comes across after being tasked with improving an Internet connection is that the connection is overutilised. Typically nobody knows why, who, or what is causing the problemexcept of course everyone blames the ISP. Sometimes it is the ISPbut typically you can’t prove that without having an alternative connection immediately available. I currently manage or help manage foursites/premisesthat use QoS to manage their Internet connectivity. One is my workplace, two are home connections, and the last one is a slightly variable oneusually just a home connection but alternatively, for a weekend every few months, it becomes a 140-man (and growing) LAN. Fun. 🙂

MikroTik and RouterOS

MikroTik‘s RouterOS is very powerful in the right hands. Many other routers support QoS but not with the fine-grain control MikroTik provides. Alternatively you could utilise other Linux-based router OS’s, such as DD-WRT, Smoothwall, Untangle, and so forth. Most of these typically require that you have a spare server lying about or a compatible hardware router. Mikrotik sells RouterBoards that have RouterOS builtinand they are relatively inexpensive.

My experience with routers is primarily with Cisco and MikroTikand my experience with QoS is primarily with Allot’s NetEnforcer/NetXplorer systems and MikroTik. The most popular MikroTik devices in my experience (other than their dedicated long-range wireless devices) have been their rb750 (new version namedhEX“) and rb950-based boards. They have many others available and are relatively inexpensive. In historical comparison with Cisco’s premium devices, I’ve tended to describe MikroTik’s devices as “90% the features at 10% the cost”. As this guide is aimed primarily at SME/Home use, inexpensive makes more sense. If you’re looking at getting a MikroTik device, note that MikroTik routers do not typically include DSL modems, thus your existing equipment is typically still necessary. Note also that this is not a tutorial on setting up a MikroTik device from scratch. There are plenty of guides available online for that already.

Theory into practicefirst steps

To set up QoS correctly, you need to have an idea of a policy that takes into account the following:

  • The overall connection speed
  • How many users/devices will be using the connection
  • The users/devices/services/protocols that should be prioritised for latency and/or throughput

To achieve the above in my examples, I will assume the following:

  • The MikroTik is set up with the default network configuration where the local network is 192.168.88.0/24 and the Internet connection is provided via PPPoE.
  • The connection speed is 10/2Mbps (10 Mbps download speed; 2 Mbps upload speed)
  • There will be 5 users with as many as 15 devices (multiple computers/tablets/mobile phones/WiFi etc)
  • Typical downloads require high priority with throughput but low-priority with latency
  • Gaming/Skype/Administrative protocols require high priority with both latency and throughput
  • No users are to be prioritised over others

The first and probably quickest step is to set up what RouterOS refers to as a Simple Queue.

I’ve made a short script that I have saved on my MikroTik devices to set up the simple queues. It is as follows:

:for x from 1 to 254 do={
 /queue simple add name="internet-usage-$x" dst="pppoe" max-limit=1900k/9500k target="192.168.88.$x"
}

What the above does is limit the maximum speed any individual device can use to “1900k” (1.9Mb) upload and “9500k” (9.5Mb) malŝarĝo.

Notes:

  • The reason why the max limits are at 95% of the line’s maximum speed is that this guarantees no single device can fully starve the connection, negatively affecting the other users. With a larger userbase I would enforce this limit further. For example, with 100 users on a 20Mb service I might set this limit to 15Mb or even as little as 1Mb. This is entirely dependent on howabusivethe users are and, as you figure out where and how much abuse occurs, you can adjust it appropriately.
  • The prefixinternet-usagein the name parameter can be customised. Typically I set these to refer to the premises name. For example, with premises namedalphaandbeta”, I will typically putinternet-alphaandinternet-beta”. This helps with instinctively differentiating between sites.
  • The dst parameter haspppoein the example. This should be substituted with the name of the interface that provides the Internet connection.

Ensure you customise the script to be appropriate to your configuration. Save the script to the MikroTik and run itor paste it directly into the MikroTik’s terminal to execute it.

In my next post I will go over setting up what RouterOS refers to as Mangle rules. These rules serve to identify/classify the network traffic in order to make finer-grained QoS possible.

Interŝanĝado
Category: random  | Leave a Comment
Thursday, September 17th, 2015 | Author:

Privacy, Time, Money

I don’t like debit orders. I’ve never liked the idea that another entity can, at will, take almost any amount of my money (wellwhatever’s available). A colleague pointed out the issue with MTN would have been avoided had I been using a debit order. Maybe theconveniencefactor isn’t such a bad thing.

I suppose the penultimate question here is whether or not you want the convenience and can trust institutions (in this case with your money) – or if you can’t trust them and are willing to forgo that convenience. In my case, even though I still question the convenience, I learned the hard way with MTN that it doubly can be inconvenient to have your connected world reduced toremote islandstatus. Almost everyone today goes with the convenience factor.

Convenience

On the other hand, now a long time ago, I had a dispute with Planet Fitness where convenience was a double-edged sword. I reported their business practice to the Consumer Complaints Commission (since re-organised as the National Consumer Commission) and never got feedback from them. The gist of the issue is that Planet Fitness’s sales agent lied to me and a friend in order to get more commission/money out of my pocket.

I’m a Discovery Vitality member which gives many benefits, including reduced rates on Premium brandsmostly health-related of course, as Discovery is a Medical Aid/Health Insurance provider. To put it simply, Discovery is awesome. Vitality’s benefits cover gym memberships which further includes Planet Fitness. You still have to pay something, a small token of sorts, to Discovery, for the gym membership. But, after all, they WANT me to be healthy, so they don’t mind footing the bulk of the bill. But, apparently, this means Planet Fitnesssales agents don’t get the commission!

So what does this result in? The result is that PF’s sales agent gave me an inflated figure for aVitality-basedmembership. He lied. He then had me sign on the dotted line for an inflated price of aregularmembership (yes, it was actually more than even a regular membership would have cost), ending up about 4 and 5 times as much as the Vitality-based membership.

Epiphanies

Some time in 2011 I finally wisened up to the costs I was supposed to be paying. Discovery I am sure wouldn’t be too happy about this fiasco. I spoke to the Manager at the gym, and I was assured that the entire contract would be scrapped. I’m not one for violenceunless its for sportin an Octagonbut after my 5th visit to the Manager to ask why the Debit Orders were still happening, he told me he was surprised I hadn’t brought weapons with me for the visit. After a few more visits, the Manager had actually left Planet Fitness and explained to me that thecontractwas between myself and Head Office and that the local gym, apparently a franchise-style operation, had little to no say about whether or not it could be cancelled. If Head Office said no, tough luck.

By this point I’d lost it. I had my bank put a stop to the debit orders. It was a huge schlep: I had to contact the bank every month because the debit order descriptions would change ever so slightly. It also cost me a little every couple of months toreinstatethe blocking service. I can’t help but think the banking system supports regular expressions but the staff don’t necessarily know how to use it.

Technically I’m still waiting on the CCC to get back to me (never happenedand of course they were re-organised as mentioned above so the case probably fell through the cracks). Of course, by that point PF also wanted to blacklist me for not paying!

The Unexpected Hero

A haphazard mention of the issue to Discovery (I think I called them about a dentist visit) resulted in a callback by one of Discovery’s agents. They then asked me to describe the problem, in detail and in writing, to better explain from my perspective what had really happened. I obliged. It turns out I was right about them not beingtoo happyabout it. In fact they really didn’t like it. About three weeks later, Planet Fitness refunded me in FULL for all monies that had ever been paid to them.

Discovery is Awesome. 🙂

Interŝanĝado
Sunday, August 04th, 2013 | Author:

I had a power outage affect my server’s large md RAID array. Rather than let the server as a whole be down while waiting for it to complete an fsck, I had it boot without the large array so I could run the fsck manually.

However, when running it manually I realised I had no way of knowing how far it was and how long it would take to complete. This is especially problematic with such a large array. With a little searching I found the tip of adding the -C parameter when calling fsck. I couldn’t find this in the documentation however: fsckhelp showed no such option.

The option turns out to be ext4-specific, and thus shows a perfectly functional progress bar with a percentage indicator. To find the information, instead offsckhelp” Aŭ “man fsck”, you have to inputfsck.ext4help” Aŭ “man fsck.ext4”. 🙂

Interŝanĝado
Sunday, August 04th, 2013 | Author:

History

Much had changed since I last mentioned my personal server – Ĝi kreskis de saltegoj kaj saltegoj (it now has a 7TB md RAID6) and it had recently been rebuilt with Ubuntu Server.

Arch was never a mistake. Arch Linux had already taught me so much about Linux (Kaj daŭros tiel fari sur mia alia surtabla). But Arch definitely requires more time and attention than I would like to spend on a server. Ideally I’d prefer to be able to forget about the server for a while until a reminder email saysumthere’s a couple updates you should look at, buddy.

Space isn’t freeand neither is space

The opportunity to migrate to Ubuntu was the fact that I had run out of SATA ports, the ports required to connect hard drives to the rest of the computerthat 7TB RAID array uses a lot of ports! I had even given away my very old 200GB hard disk as it took up one of those ports. I also warned the recipient that the disk’s SMART monitoring indicated it was unreliable. Kiel intertempa workaround al la manko de SATAaj havenoj, I had even migrated the server’s OS to a set of four USB sticks in an md RAID1. Freneza. Mi scias. Mi wasn’t tro feliĉa pri la rapido. I decided to go out and buy a new reliable hard drive and a SATA expansion card to go with it.

The server’s primary Arch partition was using about 7GB of disk. A big chunk of that was a swap file, Cached datumo and otherwise miscellaneous or unnecessary files. Entute la reala grandeco de la OS, including the /home folder, Estis nur pri 2GB. This prompted me to look into a super-fast SSD drive, thinking perhaps a smaller one might not be so expensive. It turned out that the cheapest non-SSD drive I could find actually cost Pli Ol unu el ĉi tiuj relative malgranda SSDs. Yay for me. 🙂

Choice? Woah?!

En elektanta la OS, I’d jam decidis ĝin wouldn’t esti Arkaĵo. Ekstere de ĉiuj la aliaj popularaj disdonadoj, I’m most familiar with Ubuntu and CentOS. Fedora was also a possibilitybut I hadn’t seriously yet considered it for a server. Ubuntu won the round.

The next decision I had to make didn’t occur to me until Ubiquity (Ubuntu’s instalaĵa sorĉisto) Demandis ĝin de mi: How to set up the partitions.

I was new to using SSDs in Linux – I’m bone konscia de la enfaliloj de ne uzanta ilin ĝuste, Plejparte pro ilia risko de malriĉa longevity se misuzita.

Mi didn’t deziras uzi diligentan interŝanĝan septon. Mi planas sur altgradiganta la servilon’s motherboard/CPUa/memoro ne tro ege en la estonteco. Bazita sur kiu mi decidis min metos interŝanĝon en interŝanĝa dosiero sur la ekzistanta md TRUDENIRO. The swap won’t be particularly fast but its only purpose will be for that rare occasion when something’s gone wrong and the memory isn’t available.

This then left me to give the root path the full 60GB out of an Intel 330 SSD. Mi konsideris apartiganta /hejmon sed ĝi nur ŝajnita malgranda sencela, Donita kiom malmulte estis uzita en la pasinteco. I first set up the partition with LVMsomething I’ve recently been doing whenever I set up a Linux box (really, there’s no excuse not to use LVM). Kiam ĝi akiris al la parto kie mi formus la filesystem, Mi klakis la falon-malsupren kaj instinkte elektita ext4. Tiam mi rimarkis btrfs en la sama listo. Pendi sur!!

But a what?

Btrfs (“Butero-eff-ess”, “Pli bona-eff-ess”, “Abelo-arbo-eff-ess”, Aŭ kio ajn vi fantazias sur la tago) Estas relative nova filesystem evoluigita por alporti Linukson’ Filesystem kapabloj reen sur vojeto kun fluo filesystem tech. La ekzistanta King-de-la-Monteto filesystem, “Ext” (La nuna versio vokis ext4) Estas sufiĉe bona – but it is limited, stuck in an old paradigm (think of a brand new F22 Raptor vs. an F4 Phantom with a half-jested attempt at an equivalency upgrade) and is unlikely to be able to compete for very long with newer Enterprise filesystems such as Oracle’s ZFS. Btrfs ankoraŭ havas longan vojon iri kaj estas ankoraŭ konsiderita eksperimentan (Dependanta sur kiu vi demandas kaj kio prezentas vin bezono). Multaj konsideri ĝin esti stabila por baza uzo – Sed neniu estas iranta fari ajnajn garantiojn. Kaj, Nature, Ĉiu estas diranta fari kaj testaj rezervoj!

Mooooooo

La plej fundamenta diferenco inter ext kaj btrfs estas ke btrfs estas “BOVINO” Aŭ “Kopii sur Skribi” Filesystem. Tio ĉi signifas ke datumo estas neniam efektive intence anstataŭigita de la filesystem’s internals. Se vi skribas ŝanĝon al dosiero, btrfs will write your changes to a new location on physical media and will update the internal pointers to refer to the new location. Btrfs iras paŝon cetere en kiu tiuj internaj sugestoj (Plu-gvidita al kiel metadata) Estas Ankaŭ BOVINO. Pli malnovaj versioj de ext havus simple anstataŭigitan la datumo. Ext4 uzus Revuon certigi tiun korupton gajnis’t okazas devus la AC ŝtopas esti elretirita ĉe la plej inopportune momento. La revuaj rezultoj en simila nombro de paŝoj postulita ĝisdatigi datumon. With an SSD, the underlying hardware operates a similar CoW process no matter what filesystem you’re using. Tio ĉi estas ĉar SSDaj stiradoj povas ne efektive anstataŭigi datumon – Ili devi kopii la datumon (with your changes) to a new location and then erase the old block entirely. An optimisation in this area is that an SSD might not even erase the old block but rather simply make a note to erase the block at a later time when things aren’t so busy. The end result is that SSD drives fit very well with a CoW filesystem and don’t perform as well with non-CoW filesystems.

Fari aferojn interesanta, BOVINO en la filesystem facile iras manon mane kun ĉefaĵo vokis deduplication. Tio ĉi permesas du (Aŭ pli) Identaj blokoj de datumo esti entenita uzanta nur ununuran kopion, Savanta spacon. Kun BOVINO, Se deduplicated dosiero estas modifita, La aparta ĝemelo gajnis’t esti influita kiel la modifita dosieron’s datumoj estos estinta skribita al malsama fizika bloko.

BOVINO en ĝiradaj faroj Snapshotting Relative facile efektivigi. Kiam momentfoto estas farita la sistemon simple registras la novan momentfoton kiel estanta duplikatado de ĉiuj datumoj kaj metadata ene de la volumo. Kun BOVINO, Kiam ŝanĝoj estas farita, La momentfoto’s datumoj restas sendifektan, Kaj konsekvenca vido de la filesystem’s statuso ĉe la tempo la momentfoto estis farita povas esti daŭrigita.

A new friend

Kun la supra en menso, Precipe kiel Ubuntu faris btrfs havebla kiel instali-tempa elekto, Mi kalkulis ĝin estus bona tempo plonĝi en btrfs kaj esplori malgrandan. 🙂

Part 2 coming soon

Interŝanĝado
Monday, October 29th, 2012 | Author:

It appears that, in infinite wisdom, Google have a security feature that can block an application from accessing or using your google account. I can see how this might be a problem for Google’s users, in particular their GTalk and Gmail users. In my case it was Pidgin having an issue with the Jabber service (which is technically part of GTalk). I found the solution after a little digging. I was surprised at how old the issue was and how long this feature has existed!

To unlock the account and get your application online, use Google’s Captcha page here.

Interŝanĝado