Designing a Server System for a School

I’m new to linux; coming from a Mac and Windows background, and I’d like to design an entire school network server that can handle serving 2500 simultaneous student connections as well as 100 administration connections.

I’ve looked at both DebianEdu 11.5 and LinuxSchools 14 (ubuntu based).

Though both distributions could be perfect solutions, their GUI’S; (debian with gosa2, and linuxschools with its own web based console), both are dated and would not inspire staff and students alike.

Would you be able to point me in the right direction for such an effort? My goal is to create an ‘out-of-the-box’ solution for public school systems in my region. The schools would require that only ‘one’ linux operating system be used by students, requiring the client machines to be able to run securely from various systems. i.e., windows, macs, ipads, tablets, watches, etc…

What are you trying to serve? I’m not familiar with gosa or linuxschools, are these educational servers?

If security is the concern, then I have bad news for you. I admire your efforts, but unless you run more than just one Linux box and have an admin to maintain it (or enable some automatic updates, which make the system prone to breaking when needed the most), you are not adding security, because this linux box will be the one thing neglected by the IT staff that don’t know how to update linux.

Having more than just one linux box and having at least a part-time admin managing it would come a long way though.

As for the 2500 simultaneous connections, with a dedicated server you may need a beefy spec. Assuming it is just apache and php, probably a high frequency 6 or 8 core server with 64GB of RAM. If traffic can be cached, you could only get away with less if you add reverse-proxies in-between and load-balance and cache pages and content, but this is not usual in a small school environment.

I’m a little confused about what you actually need.

It sounds like you want to create a remote desktop environment, where any device can connect via remote desktop, and be presented with a standard Linux Desktop.

I would try and keep the server and desktop parts distinct from each other.

The server part is about choosing the right hardware (if you’re hosting your server yourself), and choose the right base OS. This could be any server Linux distribution, like Ubuntu Server on proper hardware - or an externally hosted solution. LinuxSchools also looks interesting (I don’t know it).

The desktop part is about what a user is presented with when they log on with the remote desktop solution. So this is about choosing which remote desktop solution will be used, and what OS will be presented to users (where DebainEdu is possibly an option?). This can run in VM’s or even VM clusters, managed by the server(s).

Just think about splitting your project up in these 2 parts for starters, because they basically have nothing to do with each other (except the desktop solution runs on the server solution).

Absolutely correct, but this is not a typical ‘platform’ as widely available today. The current options being considered are DebianEdu 11.5+, or LinuxSchools 14+; which is a Ubuntu based distribution.

Each server ‘box’ will have 4 nodes @ 2.9Ghz per node and 2TB of RAM. They will have 4-20GB SAS 12Gb/sec hard drives. No GPU’S. (initially).The network tech in these servers are: 2 x Intel® 100-Gigabit Ethernet Network Adapter E810-CQDA2 - PCIe 4.0 x16 - 2x QSFP28 and 2 x Intel® 10/40-Gigabit Ethernet Converged Network Adapter XL710-QDA2 - PCIe 3.0 x8 - 2x QSF. That’s what I have to work with.

The debate right now with school officials is mostly about ‘tracking’ and coursework integration as a ‘lifestyle’; which includes homework. The server system will use custom software packages to not only track student progress in particular ways, but will also interactively inform department heads of a student’s decline in performance or ‘red-flags’ for culturally specific threats, so that trained mentors; health, mental-health, lifestyle coaches, etc., can reach out to help those students acquire the needed coursework skills without the stresses typical in public education environments. This ‘framework’ will include physical and mental-health considerations specific to each student as an individual—in as much as possible in a generalized way, but with consistent and targeted methodology proven to be effective. (such things as race, culture, lifestyle, background, etc., being considered.)

The security aspects; given our laws that protect privacy; health information, especially mental-health information, require strong measures that could present bottlenecks in education operations. But I digress… Sorry.

Currently, we’re debating whether to use a centralized blade server, or to deploy smaller servers at secured locations throughout the school. The latter, imho, presents much greater risks. The former, raising costs astronomically. i.e. IT staff, Security personnel, etc…

The key issue here is that cash-strapped public school systems mostly depend upon students having their own laptops or tablets. A community, middle-class families and the like, deserve that their kids, and even their adult kids, have a fair opportunity to grow and flourish. I can’t think of too many middle-class communities that can afford to buy each and every new student a specific device each year. That would cost Millions$$$.

Integrating the various machine platforms is a tough task. LTSP, as in the DebianEdu project is enticing, but that too is dated and quite ugly and would require serious modifications that “I” would have no idea where to begin. Consistent GUI throughout? No distro has that down pat, but ultimately, the purpose is education.

One server ‘box’ ; as mentioned above will not serve 2500 simultaneous connections, each operating at potentially, full throttle, given the heavy use of interactive video, pop-in-apps, etc. Time is of the essence in education, but ‘life’ happens too; as I’m sure you know, and education cannot be done successfully solely from a computer screen for 7 hours each day. It’s unhealthy. We’re integrating hands-on training with proper use of technology throughout the process.

A federated systems approach was mentioned for the distributed ‘box’ approach, (load balancing?), but I tend to think that THAT, could introduce a potential nightmare in stability, for security and updating staff. More pipelines, more wires, hardware and labor. That also means more parts that breakdown. More future expenses that public-schools never have.

Simple plug and play access, lightning speeds, and extensibility is what I’m looking for.

Any ideas?

Ok, got the gist of it. While I don’t agree with tracking (and I’d probably get my kids out of the public education system anyway), I see your requirements, so I will focus on that.

Unfortunately, I’m not sure how the application works, so I can’t comment much on the decentralization aspect. If possible, I would set a server on each campus, because it reduces latency and distributes the load. Using a different DNS server on each campus can allow the site to be loaded from each individual server. It increases complexity a bit if the infrastructure is not already in place, but should help with redundancy, localization and latency, the last which I find most important.

But if there is also a move to have more schooling from home options, then you’d need the ability to access the sites from the internet and know which one is which. That changes how things are configured, it might be better off to do it from a load-balanced, internet-accessible reverse-proxy and have each campus have a different URL, while using the same FQDN (like say, etc.).

Having it organized like this probably allows for a bit of flexibility and doesn’t require students install stuff on their personal devices.

A problem with old platforms is that they may not be adaptable to different input devices. For example, my nieces did not have computers until middle school and high school respectively, they had to use either their tablets, or their mom’s laptop. The proprietary platform allowed adaptable interfaces on their phones, but will the software that you are using allow for that?

In case of children, accessibility should not be taken for granted, because not everyone can use a desktop browser.

Back to the setup, I would say that a central server’s only advantage is that… well… the data is centralized, so it makes things easier, in the sense that you don’t have to migrate the data from a campus to another if a kid moves to another school. But with a distributed system, manual migration of the data may need to happen, which means someone has to do that, so it is an added operational cost. I don’t know the statistics on how often kids change schools, but you should expect at least a couple dozens a year.

If you have a staging server with similar data on it, you should be able to update it first, make sure the update is successful, then push the update to the rest, all through a workflow. Updating is not a hard thing to do once you have the setup in-place. Even an update script and a pssh can get you a long way, but it is important that the update is tested first. This does not matter if it’s a central system or distributed boxes.

It might be harder to monitor each box, which is why even more complexity is needed, by adding a monitoring server like Zabbix, but it should improve security, not decrease it. Imagine what would happen if a hacker breached the centralized server and got all the data. At least by having a distributed one, at least you mitigate the damage just to a specific campus. But a monitoring system should be used for a central system too, just that it could be something easier to set up, like monit or down detector.

For security, as long as you implement the 4-5 typical web security options, you should be fine:

  • Set a very complex admin password (use password managers)
  • Enable 2FA if applicable
  • Change the management page to a 32 character random page in the server config
  • Restrict access to the management page from certain IP addresses, like from inside a VPN or a private management campus network
  • Rate-limit password guessing with something like fail2ban, iptables or crowdsec
  • Bonus, disable password login on SSH and only allow key-based auth, with password-protected keys for users and just a key for the automation system (for the updates)

Note that the load-balanced reverse proxy can be something like Cloudflare.

Thanks. That makes a lot of sense. So, I watched the video on ANSIBLE. And got a few ideas for backup systems that could be independently reinstalled; if the operating system flies south.

I remember an app called Deep Freeze, by faronics, that preserved the operating system in prestige form, and would only permit files to be changed in specific directories; such as a user’s home folder, and I remember that I never had a problem with my OS. Can Linux do that at the server level?

I need the ServerOS to be indestructible. The best way for me to describe it is that the os be read only, and changes can happen to only files is a specific directory; i.e., user/home-folder.

Oh, btw: my use of the word ‘tracking’ may have been misconstrued. In that case, the tracking was solely about app-use, and the interaction with it. Period.

So I guess that one of the boxes as described earlier would be better situated by department/class level, where each class would be a site with its own IP. DebianEdu seemed to do that out of the box, and it’s streamlined. In this case, what kind of backup system could be used to assure user home-folder integrity? A daily snapshot held on a backup server?

I used to use Reboot Restore Rx on Windows, it did a good job.

For the server, well, it depends what you mean. Normally you don’t want to restore the system to the previous state, because that means updates will not be applied. Also, Linux has a different file system hierarchy, so it is likely that restoring to a different point in time will wipe some of the user data. You can of course exclude the home directory, but usually, servers keep their data in places like /var/lib/app-name (like /var/lib/mysql/).

On Linux, there are way better solutions for this, but it depends on your level of expertise. OpenSUSE has snapper that does a snapshot of the rootfs partition before each update, so you can recover to it with a reboot if an update breaks stuff. It even creates an easily selectable menu entry in the bootloader to pick the desired snapshot to revert to. It works by leveraging the snapshot engine from BTRFS.

Another indestructible rootfs is by using something like Fedora Silverblue with atomic backups, but this is mostly recommended for other things, like flatpak and docker programs. I would not recommend this.

Two other noteworthy mentions are NixOS and Alpine diskless install. NixOS which creates a read-only store (aptly called /nix-store) where all the programs are installed and which basically does a cold-freeze of the entire system without using snapshots. Of course, you can leverage snapshots with it, but it’s not necessary. You can basically opt-out of all the file system hierarchy except /boot and /nix-store and nix can automatically recreate the FS for you on boot. You can read more about it here.

Alpine has a different method. A diskless install means that you are booting an OS on a ramdisk, making it smoking fast and deleting any data that is not explicitly backed up using lbu. If can increase the lbu retention and if you mess something up, you can boot a live environment and change the system state and data to be loaded from an older lbu backup file. Although not as versatile as nix and both “suffer” from booting into a new initramfs and kernel image if it is updated (well, all linux systems do, but this can be changed from a bootloader like GRUB, so I would not call this a weakness to any linux distro, including NixOS and Alpine).

I was thinking more in the campus level. Say that you have 1 school with 3 campuses, each campus would get its own server. If kids transfer between campuses for any reason, the data should be moved over to the other campus. But if the kids change the campus daily, that would be a big problem, because you can’t just ask someone to move data from a place to another willy-nilly, so a central server would make more sense. The architecture needs some serious planning.

For backups, I am a big fan of ZFS. You can take a snapshot of the file system and do a zfs-send to another server, even in another locatio and you can do so incrementally, so data transfer is minimal. And with deduplication, it can technically be possible to have more than just one full backup and a few incrementals with just barely above the size of the current resource consumption.

In absence of ZFS, which can be a bit tricky to set up if you are not using a distro that comes with it, like proxmox or ubuntu (ubuntu desktop, I don’t know if ubuntu server has a zfs option yet), you can still use rsync to synchronize the home folders incrementally.

Note that if this web server uses MySQL, Postgres, or any other DB, the backups for DBs are a bit different and you can’t just backup the FS while the DB is running, that will leave you with inconsistent (useless) backups, so you need to use utilities like mysqldump, percona xtrabackup, pg_dump and others. Also, DBs don’t like snapshots, unless you have a mechanism to take them (a script to log into the DB, lock all the tables, take the snapshot, then release the locks, which kinda impacts the DB for a short period).

Many admins tend to stay away from mixing server distros. How do you feel about that?

Correct me if I’m wrong, but I’ve included a peanut gallery image of what I believe you’re describing? Is my interpretation correct?

Well, this map is definitely not explaining much, especially since the switches are drawn kinda like routers. Also, why is the DMZ drawn as a server, beats me.

But leaving that aside, with the main server in one place and each campus having a server, you are more describing some kind of reverse-proxy as opposed to the distributed system I would be thinking of. Again, I have no idea about the server architecture of the software you plan to use or if it even supports a distributed model.

But even with a central server and a couple of reverse-proxies in each campus, it should serve as a decent architecture and not overcomplicate the setup. And if the main site can be cached (which the reverse-proxies should be able to do), then all the better.

But for 2500 people, I would have a AAA server somewhere (Samba OpenLDAP, or if it’s already in place in the school, Active Directory) and have an AD DC (Active Directory Domain Controller) in each campus. Technically speaking, a single server should be able to handle most of the tasks, if separated as VMs or containers. For redundancy of the server itself, you could use something like OS clustering like corosync + pacemaker and if a campus server just goes out, redirect the server to one of the other campuses, or the central server. Might make the experience a bit slower, but better than just having an entire campus idling.

I also tend to do that, but as long as you have a decent upgrade plan, it should not be a problem. You could make an update script which just runs the typical package manager update commands and execute it to all servers, no matter the distro, using pssh. Of course, if you use Ansible, that task is already basically done, you just need to learn ansible (which is not terribly hard, just different).

In the previous environment I managed, I used CentOS and Oracle Linux (and some Scientific Linux here and there). Most of the servers weren’t set up by me, but they were there already. Since they were mostly RHEL based, we continued with CentOS. We had versions from 5 to 7 and I started doing some 8 (before RHEL killed CentOS 8 in favor of Stream). But I did have 2 or 3 Ubuntu VMs, running things like OCS Inventory, that was much easier to set up as such. And we had some very old Ubuntu routers that we just kept on using until somewhat recently when my ex-colleagues got the budget to upgrade.

With a decent upgrade plan, I would not shy away from using different distros and even different OS (like say FreeBSD for NASes).

My current homelab has a FreeBSD NAS and Void Linux servers (I would not recommend using Void in production and even at home, unless you know what you are doing, just use popular distros with many users, like Rocky Linux, Fedora or Debian). I am planning for an OpenBSD router once drivers are available for my USB WiFi card (or I just get internet another way and get rid of the dongle, my router currently runs Void too, my previous router used to be a RPi 3 running Alpine, which was only used because it was running diskless and saving my SD card, but this router has eMMC now, so I don’t care anymore).

I’m not sure if this forum is the best place to ask these questions. Well, in all honesty, without knowing the inner workings of the software you plan to use, I cannot really say how to implement all these stuff.

I cannot find much information about DebianEdu, it looks like a distro with software preinstalled. As for LinuxSchools, it looks like a suite of software bundles, like Samba and Samba AD DC (basically emulating Windows AD), email (dovecot and probably postfix), Moodle (for e-learning, I think this is the software you may actually need out of everything provided), DHCP, OwnCloud (file sharing), Joomla (website, eeww, but I guess should be normie-friendly, kinda like a slightly-better wordpress and not as insecure), nginx, glusterfs and a few other niceties.

An experienced linux admin can set up most of these and automate their updates and maintain these pretty easily, but not sure about the end-users (students) and the platform-admins (people who are supposed to add kids in the system, give access and so on).

Giving a read, I do not know exactly what the architecture of LinuxSchools is, but I can confidently say, you should not run this thing in different locations. You need to run them in a single datacenter / server room. You can have some localized access, like the aforementioned reverse-proxies (if Moodle can be cached) and AD DCs, but most of the heavy duty will be pulled by a few servers in a data room, following a somewhat centralized system.

1 Like

Thank you so much for taking the time you have. Your youtube channel is phenomenal, and the fact that you include your reasoning in your presentations makes your channel a step above the rest.

I am now convinced that a centralized server might be our best option, as you suggested. Cable costs, router costs and switches are going to increase, but ultimately, we’ll save $$$ with a centralized system.

I checked out Alpine Linux, and found it very portable, and may come in handy here and there. I’ve yet to look at nixOS, but I will later this evening.

I’d like to encourage you to do a tutorial/setup of either linuxschools server, or debianEdu. Your well organized presentation of the subject, with your reasoning behind the mission makes a huge difference that goes well beyond the typical youtube presentation. Kudos!

I’m an old guy, almost 70, and when computers first began at the university, I was feeding punch-cards into a scanner, lol. I also put together RAM trays, where each memory core had to have an x-wire, y-wire and a sensor wire fed through to the next milliscopic core. All day under a microscope turned me off to computing for a bit. The server room I worked in was an entire 4 story building. Today, my iPhone does more than that entire building did.

Once again, many thanks for your advice. You presented ideas for me that I never would have discovered without your help.

1 Like

That is cool and all, I appreciate your kind words, but I’m not @jay, lmao, just a rando on the forum.

NixOS is a beast of its own, but the nix package manager is portable to any other Linux distro and even macOS, but NixOS has a few tricks up its sleeve with the way the file system binaries and configurations are immutable (the /nix-store is a read-only FS for the users and files on it should only be modified by nix, not manually by hand).

But yeah, like I said before, Ubuntu and CentOS are probably the best bet, with linuxschools being based on ubuntu, so should be pretty easy to use, while Jay’s ubuntu book can also come in handy.

1 Like