Today, I got the news that a co-worker of mine passed away. She was one of the first people I on-boarded at this client when I previously worked for them as a tech with an MSP and not as their manager. She was always smiling and made me feel so accepted at a place that I was unfamiliar with. She was a mother, under 50 yrs old and had a young daughter she was co parenting. I would get to see her daughter often at the org and you could tell how much her mom loved her.

Now, she is gone and I have to retrieve her laptop. Maintenance has to clean up the office and we have to move on.

It is the second person this year that I have closely worked with that I will have to terminate and it is devastating.

I know this is a somewhat common post here, but I think it's a conversation that is very important for those of us that have had to deal with grief in the workplace.

She had a direct impact on my experience at work and me feeling like I was at the right place in my career.

I want her memory to live on so I will tell others of our interactions and how sweet she was.

If you have had to go through similar, please feel free to share your own experiences with this.
Sending love to anyone dealing with death.

I just wanted to get this off my chest. I don't have anyone to talk to who would understand what I'm going through.

I'm in hell. The metrics are unattainable, and there are too many tickets. I'm getting anxious every day, and I've even started dreaming about tickets. My managers call me out almost every week because I miss a ticket.

I want to leave this job, but I can't for now.

As if Microsoft and their software/cloud isn't bad enough in it's current state, it's likely going to get worse with a reduction of about 1/4 of their workforce.

Microsoft makes new 'voluntary exit' offer to about a quarter of its workforce, including some execs – GeekWire

I'm eager to see how many more shit updates get through MS "QA" into production environments and cause unnecessary havoc and stress globally.

Ran into this while chasing a bug report and it surprised me, so sharing.

Spamhaus doesn't answer queries that come through big public resolvers. Google returns an empty answer, which your software reads as "not listed". So a Postfix reject_rbl_client zen.spamhaus.org on a box that resolves through 8.8.8.8 lets everything through, and nothing ever logs an error.

10-second test, run on your mail server:

dig +short 2.0.0.127.zen.spamhaus.org

That's Spamhaus's permanent test entry; it should always return 127.0.0.2. Empty or 127.255.255.x means your lookups aren't working.

Fix: run a local recursive resolver (Unbound takes five minutes) and point the mail server at it. If you're non-commercial, a free Spamhaus DQS key also works.

How I found it: I maintain a small self-hosted blacklist monitor, and a user reported an IP that was on Spamhaus but showed clean. Same cause. It now tests every list against its test address first and shows "no answer" instead of a fake "clean".

Curious how common this is: what does the dig return for you?

Is anyone else's team still recovering from the recent PHP2Shell fire drill (CVE-2026-109618 / 109619)?

How we got here is what scares me the most. The developers knew about a memory issue back in 2024. They ignored it to keep older systems working, thinking no one could actually use it to hack into systems.

But then, someone used a new AI model on the old C codebase. The AI quickly found a complex bug that humans had missed for years. To make things worse, a software company accidentally released the fix 12 hours too early. Because of this mistake, hackers could see exactly what was changed. They started attacking the internet before the official warning even came out.

For 20 years, old open-source code stayed safe mainly because finding memory bugs by hand is incredibly hard, boring work. Now, AI models can easily find hidden flaws in 15-year-old code. Volunteer developers simply can't keep up with this speed.

How are you all handling this at work? If we now have to fix servers in four hours or unplug them completely, I might just move to the woods and raise goats.

I created a new email address with a public and free mail service and sent an email to HR asking for my salary to be transferred to a new bank account. Surprisingly, they didn't ask for any verification on my corporate email address, private mobile number or email address that I provided on the paper form when I was hired, and proceeded with the change.

To spice things up, I:

• thanked them for the self-arranged countryside holiday bonus, the details of which are available on job posting platforms

• used the full corporate external signature containing my name, surname, position, address details and corporate logo, all of which can be found on our webpage.

I will need to wait until the next salary transfer to verify whether it has actually been paid into the other bank account I own. Then, I will notify the Security Team about the incident and arrange a meeting with my supervisor and HR.

Do you have any advice on how to ensure this doesn't backfire on my manager, HR team and me?

... and rats are gnawing the floorboards.

(To be kinda meta, this post isn't seeking anything except commiseration and conversation. I'm just reflecting on the dizzying array of changes within the last ~3 years, and hoping a graybeard out there can relate to feeling ... I don't know, displaced? surreal? anyways, my apologies if this doesn't belong in r/sysadmin*)...*

While the underlying tech has evolved, the core of the Internet and computing remains the same as it was in the 1980s: content is stored on hosts and delivered to clients through multiple layers of protocols.

It makes sense, then, that AI will discover uncontrollable amounts of vulnerabilities. The visionaries who initially drafted the RFCs for DNS, HTTP, and TCP/IP couldn't have possibly imagined our data getting commodified, names and birthdays sold for pennies or held for ransom, financial ruin inevitable. Of course they baked their naiveties into the tech. Of course history is getting sandwiched between OpenAI and Anthropic, with attendant freakouts from suits and normies alike.

It seems like the wooden planks are decaying, but the replacements don't fit anywhere yet. Maybe it's time we transition away from the existing infra and replace it in its entirety with the Semantic web and MCP/agents. Give up the whole ghost.

Can anyone relate? Does this resonate at all?

Anyone else seeing issues with Microsoft Teams this morning? Messages not being sent or received?

Edit: Thanks for confirming everyone. GCC here as well. Tested from external cell phone as well to make sure it wasn't our internal network but wasn't seeing anything on the M365 service health page.

Been reading up on the EWS retirement because we’re still sorting out GAL sync for our locations and I don’t want to end up with something that depends on it.

My understanding from MC1227454:

From Oct 1 Microsoft started turning EWS off in tenants that didn’t set up the app allow list. It’s phased so not everyone got hit on day one
You can turn it back on until April 1, 2027
After that it’s gone for good, no re-enabling

For those of you who already ran into it:

What broke? Was it mostly the stuff you expected (backup, sync, calendar tools) or random things nobody knew about?

How did you find what was still using EWS? Or did you only find out when something stopped working
Did you set up the allow list to buy time or just replace things?

And if a contact or calendar sync tool broke on you, did the vendor already have a Graph version ready or were you stuck waiting?

Side note for hybrid people, from what I’ve read Entra Connect also has an April 7, 2027 deadline to switch to application based auth.

I'm trying to transition into IT and would appreciate some advice from people who have made a similar move.

I have an upcoming interview for a Customer/Hardware Specialist position at a dental imaging company. It's a six-month contract-to-hire role paying $17/hr, onsite, Monday through Friday.

The job involves Level I support calls and emails, troubleshooting hardware and software issues, working with medical imaging equipment, installing equipment, training customers, documenting issues, and escalating problems when needed. My background is in education technology, where I've supported devices, printers, software, and end users, but I haven't worked in a dedicated IT support role.

I'm planning to work toward CompTIA certification and stack others.

My situation is that I have about 2.5 months of unemployment benefits remaining at $584 a week, and I'm currently living with family without paying rent. At $17/hr for 40 hours, the job would pay $680/week before taxes, so the financial difference isn't huge.

I'm trying to decide whether to take an opportunity like this to get actual IT experience while studying for certifications, or use the remaining time to study and look for a remote job paying around the same amount or more.

I think the hands-on experience could help me qualify for better IT jobs down the road. Another concern is that a six-month contract could limit my flexibility if a better opportunity comes along, and working onsite might make it harder to study.

Thanks in advance for any insights you guys have!

We've recently been seeing MS push out the new unified Copilot app, replacing the M365 Copilot app on some of our systems. Seems like this actually comes via Edge Update service of all things.

We've put the PauseCopilotUnification registry key in now, so hopefully we don't get this occuring on any more systems for now...

However, once the Unified Copilot app has been installed, users are no longer able to run it. Watching task manager, CopilotApp.exe starts & then closes immediately.

With help from ChatGPT & ProcMon, we found that it was trying to load copilot_overlay_strings_en-AU.pak and failing to find it (its not there in the install directory).
Copying the en-GB file & naming it for en-AU seems to allow it to launch successfully...

Has anybody else seen this or something just unique to our environment?

Edit: Also just found another different case where the system has ended up with both installed. On these, 'Copilot' launches however just shows a 'something went wrong' error. The en-AU.pak workaround does nothing to fix these. The exe for these versions is also 'mscopilot.exe' rather than 'CopilotApp.exe'

I'm very experienced in Intune, don't know much Jamf yet, so this is a new platform for me. Just wondering if it's beneficial to use it in an environment that only has a handful of Macbooks?

Hey everyone!
I’m building a home lab as part of my journey into IT. My current project focuses on Windows security hardening, Docker security, centralized monitoring with Wazuh, and testing whether security controls actually work. Long-term, I’d like to move toward cybersecurity/DFIR, but my immediate goal is to get into IT support or system administration.
I’d also like to build an Active Directory lab, but I’m not particularly interested in just installing Windows Server, creating a domain, adding users, configuring a few GPOs, and calling it done.
I’d like to build something that resembles a small real-world IT environment and gives me actual problems to solve. Something involving troubleshooting, permissions, security misconfigurations, authentication issues, monitoring, incident investigation, or automation.
For those of you working in sysadmin roles, what would make an AD home lab genuinely useful and interesting from a real-world perspective?
Are there any projects or scenarios you’d recommend that go beyond following a setup guide and could demonstrate practical skills to a potential employer?
I’d appreciate ideas based on things you’ve actually encountered in production environments.
Thanks!

​

Hi everyone,

I'm seeing intermittent high CPU usage from Microsoft Defender Antivirus ("MsMpEng.exe"). CPU usage reaches around 90–200%, and the alerts usually resolve themselves after a few hours.

No active scan was running during troubleshooting, and Defender appears healthy. I suspect real-time scanning triggered by application file activity may be involved.

Has anyone experienced something similar? What would you recommend checking first to identify the root cause?

Thanks!

We are seeing a increase reports of an error "Something went wrong" after logging onto any Microsoft app. Windows 11 26H2 running the latest Windows Updates.

Found this post where it was suggested KB5127648 was the cause.

Anyone else?

I've been an IT generalist at a large tech company for a few years and I will be "next in line" for a SysAdmin role soon, however, I don't think the stress and responsibility is worth the reward and opportunity.

I clearly am complacent on some level, but I am still challenged enough in my day to day that I stay engaged and interested while working. I make ~$115k/year gross including bonus, with ~5% raise yearly. I am in HCOL area but feel I am compensated well. I like my team and I like my company, which I think is rare.

In my late 30's, have bachelors in IT, almost 10 years in the field, a couple of basic IT and networking certs. I am married and don't plan to have kids. I have hobbies I am passionate about, but I have zero interest in a home lab or spending free time learning new IT skills. The extra income would be nice, of course, but the trade-off's are real, and unfortunately I don't think I can go back to my current role here if I make the jump. Pressure from management to move up is real.

I'm sure folks can relate - what do you think?

Hi all,

I'm migrating a Windows Server 2019 DHCP server (~200 scopes) to a new Windows Server 2025 server.

I exported with Export-DhcpServer -Leases on the 2019 server, but Import-DhcpServer on 2025 fails with:

"Failed to set server configuration on DHCP server ... Network Access Protection is not supported by the server. (20138)"

NAP was never used on the old server, but the export XML still contains NapEnabled / NpsUnreachableAction (server level) and NapEnable / NapProfile (scope level). Removing these elements from the XML breaks the schema validation ("The element 'IPv4' has invalid child element 'ActivatePolicies'. Expected: 'NapEnabled'").

Options I'm considering:

1. Import scope by scope with -ScopeId in a loop and move the server-level settings (option definitions, classes, server options, policies, filters) separately.

2. Skip the XML entirely: create a DHCP failover relationship from the 2019 server to the 2025 server with all scopes, let it replicate, then remove the relationship.

Questions:

- Has anyone hit this? Which approach did you use and did anything get lost (server options, vendor classes, policies, DNS credentials)?

- Is there a supported way to make Import-DhcpServer ignore the NAP settings?

- Any gotchas with the failover method for ~200 scopes (time, lease state, reservations)?

Thanks!

Anybody else's Microsoft Bookings page not correlating with their outlook calendar? This is causing HUGE issues right now with bookings appointments when people aren't even available or took vacation. I have already made a ticket with Microsoft and they are 🤷🤷🤷🤷🤷. Yes the check box for "bookable when staff are free" is checked.

Hi everyone, I'm handling a Google Workspace to Microsoft 365 Business Standard migration for 18 users and would appreciate some advice.

A few questions:

1. Large mailboxes: Five users have 100–150 GB mailboxes. We want to migrate emails from 2023 onward, but EAC doesn't seem to support date filtering. What's the best approach? Do we need a third-party tool or Microsoft's Large Mailbox Onboarding method?
2. Shared mailboxes: We have two Gmail-only accounts (info@ and billings@) that we want to migrate into M365 shared mailboxes. Is it best to license them during migration, convert them afterward, then remove the licenses if they're under 50 GB?
3. Domain and Google accounts: Our domain is managed through the old Google Domains/Squarespace setup. Should we leave the registrar unchanged during migration? What happens to existing Google accounts and domain access when we eventually cancel Google Workspace?
4. Cutover: We're planning 2–3 migration batches, starting with the shared mailboxes, followed by a weekend cutover. Any tips for minimizing downtime and preparing a rollback plan?

I'm handling the technical implementation myself, so I'd appreciate any practical advice or lessons learned. Thanks!

For sysadmins who have NinjaOne in their organization, have you had the opportunity to use Katana AI?

Our team recently acquired it, but we’re struggling to identify a practical use case for it. None of us are convinced that it’s truly “agentic” yet.

If you have any suggestions on how to effectively utilize Katana AI, please let me know!

Hello everyone,

We are planning to expand our current tape infrastructure by adding an expansion module to our existing LTO-8 tape library.

Our current setup consists of 2 tape drives and 40 slots. After the expansion, we will have a total of 4 tape drives and 80 slots. The tape drives use Fibre Channel (FC) to transfer data from the primary server.

Reason

Our RTO and RPO have increased significantly, as we are now transferring around 200 TB of data per week. We need to improve them.

I also have the feeling that our current logical infrastructure design may not be the most efficient. We are currently using Veeam Backup & Replication to perform our backup-to-tape operations.

Current

We currently have a single Standard Media Pool containing all our tapes. None of them are WORM tapes.

We use a single tape job with no incremental backups, which copies only the weekly full backups to tape. Parallel processing is enabled in the job, allowing it to use both LTO-8 drives simultaneously. The tapes are labelled both logically and physically according to their retention category: weekly, monthly, and yearly. We also label each tape with a DIMO. We do not use quarterly tapes.

Our current process is entirely manual. We select and insert different tapes depending on the occasion, rather than having an automated process. To be honest, I really dislike our current labelling system. All tapes have the same retention settings in Veeam. We manage retention manually by moving the tapes and storing them physically for the required period.

Proposed

Since we will have 80 slots available, I was thinking about using the additional capacity to keep the weekly tapes and, potentially, future daily tapes loaded in the library, as these will be the tapes we use most frequently. Our weekly tape rotation requires a total of 64 tapes, so this would allow us to keep them in the library instead of having to move around 16 weekly tapes to our off-site vault.

However, this raises my first question: Is it recommended to leave tapes loaded in the library when they are not actively being used?

I understand that these are not WORM tapes and that, in the event of a cyberattack, they could potentially be exposed to overwriting. One idea I had was to use the physical write-protect switch on the tapes to prevent them from being overwritten while they are idle.

On the other hand, I have also considered the risk of a physical incident in the data centre, although we consider the likelihood of a fire or flood to be relatively low.

Is this a reasonable approach, or is it strongly recommended to remove written tapes from the library and store them in an off-site vault?

For those managing larger tape libraries with many slots, how do you typically handle this? Do you leave tapes stored in the library and use the available slots as intended, or do you still remove written tapes manually, as we currently do?
From a Veeam configuration perspective, I am considering the following changes:

• Media Pool: Use a single GFS Media Pool containing all retention types, excluding quarterly tapes.
• Tape labelling: Eliminate our current logical and physical labelling system and rely entirely on barcodes to identify the tapes.
• Tape job configuration: Keep parallel processing enabled and configure the job to use all four tape drives.
• Media Vault: Set up a Media Vault in Veeam to better organise and track the tapes stored outside the library, as I understand this functionality is available in Veeam.

These changes would be implemented progressively as our tape operations evolve, rather than all at once.

Do you have any recommendations regarding this approach? Are there any potential pitfalls or configuration changes that you would advise against?

Thanks!!

ssh sends no keepalives of its own by default (ServerAliveInterval 0), and Linux waits two idle hours before the first TCP probe, so a script on a frozen host can hang for hours. Set the interval to 15 and a 45-second network blip kills the session. Do you tune keepalives, run commands detached and poll, or wrap them in timeout?

The certificate policy has changed, and SSL certificates will expire in 47 days. Currently, we log in to GoDaddy to renew the certificates, run a script to copy the renewed certificates to the local machine, and bind them to IIS sites across the web servers.

Can we automate certificate renewal through GoDaddy? Alternatively, are there any better options to automate the renewal and deployment process?

I've been messing around with my AD lab today and finally got one issue sorted out. I have two Windows Server domain controllers, but DC01 is currently powered off, so I'm working with DC02 for now. While running dcdiag, I found that the _msdcs delegation was still pointing to DC01. I checked the records and changed it to point to DC02 instead. Ran dcdiag again, and the DNS tests are now passing. I also jumped onto my Windows 11 client to see if it could reach DC02. DNS resolution works, ping is fine, and TCP tests to ports 53, 88, and 389 all succeeded.

The part I'm still trying to understand is that repadmin/showrpl reports replication failures involving DC01, and DC02 is still logging Event ID 2092 about FSMO role validation. All five FSMO roles are assigned to DC02, but apparently that doesn't mean everything is healthy. I'm not planning to force anything or clean up DC01 until I decide whether I'm bringing it back or retiring it properly. For those who've dealt with this before, would you focus on the Event 2092 warning first, or sort out the DC01 situation before going further?

I'm still learning the deeper side of AD, so I'd appreciate hearing how you'd approach it.

My environment uses 3 tenants to manage users. The tenants talk to each other most users are created cloud-side & we have 3 different domains. I removed a user from one of our tenant months ago. Today, i discovered their Exchange contact was still there. Okay, fine, I can remove the contact. But WHY is it still there? Where did it come from? What else is still hanging around that I don't know about?

Microsoft gives you multiple administrative views of connected objects, but no reliable, end-to-end view of an identity's origin, relationships, & lifecycle. Im expected to understand how all these systems interact, often by discovering the connections through troubleshooting.

The address book. You open a contact & theres no obvious indication of where the object came from, which tenant it belongs to, or what its associated with. Guest vs Member, cloud vs on-prem, cross-tenant identities... removing one object doesnt necessarily clean up the others.

I want to be able to look at my environment & understand where an identity came from, what its connected to & what will happen if I delete it. Apparently thats too much to ask without becoming a part-time Microsoft archaeologist.

Thanks for coming to my TED Talk. Fuck.