LIMS Troubleshooting Procedures and Processes

IIS Configuration Analysis (No Code Access)

1. Application Pool Investigation

Check app pool settings that commonly cause LIMS issues:

# View app pool configuration
%windir%\system32\inetsrv\appcmd list apppool "YourLIMSAppPool" /config

# Key settings to check:
- processModel.idleTimeout (should be 00:00:00 for LIMS)
- processModel.maxProcesses (usually 1 for LIMS)
- recycling.periodicRestart.memory (set appropriate limit)
- recycling.periodicRestart.requests (often disabled for LIMS)
- processModel.loadUserProfile (true for file access)

2. Web.config Analysis Without Source Access

Read web.config for clues:

# Find web.config files
dir C:\inetpub\wwwroot\*.config /s

# Check connection strings (redacted for security)
findstr /i "connectionstring\|database\|server=" web.config

# Look for custom errors settings
findstr /i "customerrors\|compilation\|debug=" web.config

# Check authentication mode
findstr /i "authentication\|authorization=" web.config

3. Virtual Directory and Application Mapping

Verify application structure:

# List all sites and applications
%windir%\system32\inetsrv\appcmd list app

# Check virtual directories
%windir%\system32\inetsrv\appcmd list vdir

# Verify physical paths
%windir%\system32\inetsrv\appcmd list app "Default Web Site/LIMS" /config:physicalPath

Database Connectivity Troubleshooting

1. UDL Test (Universal Data Link)

Create test connection without LIMS:

1. Create new file: test.udl

2. Double-click to open Data Link Properties

3. Configure same settings as LIMS

4. Test connection to isolate database issues

2. SQL Server Logs Analysis

Check SQL Server error logs:

-- If you have SQL access
EXEC xp_readerrorlog 0, 1, 'login failed', 'LIMS'
EXEC xp_readerrorlog 0, 1, 'timeout', 'LIMS'
EXEC xp_readerrorlog 0, 1, 'deadlock'

3. Network Connectivity Tests

Test database network connectivity:

# Test SQL Server port
telnet sql-server-name 1433

# Test name resolution
nslookup sql-server-name

# Check firewall/connectivity
ping sql-server-name

Performance Monitoring Without Code

1. Performance Counters Setup

Key counters for LIMS applications:

# Create custom counter set
typeperf -qx | findstr /i "w3wp\|sql\|memory\|processor"

# Monitor specific counters
typeperf "\Process(w3wp)\% Processor Time" "\Process(w3wp)\Working Set" -si 10 -sc 360 -f CSV -o lims_perf.csv

Critical LIMS counters:

• \Process(w3wp*)\Working Set
• \Process(w3wp*)\Handle Count
• \ASP.NET Apps v4.0.30319(*)\Requests/Sec
• \ASP.NET Apps v4.0.30319(*)\Request Wait Time
• \Web Service(*)\Current Connections

2. Failed Request Tracing (FREB)

Enable detailed tracing for errors:

1. Enable Failed Request Tracing in IIS Manager

2. Configure tracing rules:

- Status codes: 500-599, 400-499

- Time taken: >30 seconds

- All requests for specific URLs

3. Analyze XML trace files in: %SystemDrive%\inetpub\logs\FailedReqLogFiles

3. ETW (Event Tracing for Windows)

Capture detailed IIS events:

# Start ETW trace
logman create trace "LIMS-IIS" -p "Microsoft-Windows-IIS-Logging" -o C:\temp\iis-trace.etl -ets

# Stop trace after issue reproduction
logman stop "LIMS-IIS" -ets

# Convert to readable format
tracerpt C:\temp\iis-trace.etl -o C:\temp\iis-trace.csv -of CSV

File System and Permissions Analysis

1. LIMS File Access Patterns

Common LIMS file locations to monitor:

# Check temp directories
dir C:\Windows\Temp\*.* /o-d | head -20
dir %USERPROFILE%\AppData\Local\Temp\*.* /o-d | head -20

# Check upload/processing directories
dir C:\inetpub\wwwroot\uploads\*.* /o-d | head -20
dir C:\inetpub\wwwroot\temp\*.* /o-d | head -20

# Look for large files that might cause timeouts
forfiles /P C:\inetpub\wwwroot /S /M *.* /C "cmd /c if @fsize gtr 104857600 echo @path @fsize"

2. Permission Analysis

Check application pool identity permissions:

# Find app pool identity
%windir%\system32\inetsrv\appcmd list apppool "YourLIMSAppPool" /text:processModel.identityType

# Test file access as app pool user
runas /user:IIS_IUSRS cmd
# Then test file operations from that context

Network and Integration Monitoring

1. LIMS Integration Points

Common integration failure points:

• Instrument connections: Check TCP/UDP ports, serial connections
• File watchers: Monitor input directories for data files
• Web services: Test SOAP/REST endpoints
• Email notifications: SMTP connectivity and authentication
• Document management: File server connectivity

2. Port Monitoring

Monitor network connections:

# Check active connections
netstat -an | findstr :80
netstat -an | findstr :443
netstat -an | findstr :1433

# Monitor specific process connections
netstat -ano | findstr [w3wp-pid]

3. SSL/Certificate Issues

Common LIMS certificate problems:

# Check certificate store
certlm.msc

# View certificate details
certutil -store MY
certutil -store Root

# Test SSL connectivity
openssl s_client -connect yourserver:443 -servername yourserver

Error Pattern Recognition

1. Common LIMS Error Signatures

Look for these patterns in logs:

• Sample upload failures: 413, Request Entity Too Large, file size errors
• Database timeouts: timeout, deadlock, connection pool exhausted
• Session state issues: ViewState, session timeout, authentication redirects
• Integration failures: API timeouts, instrument communication errors
• Memory leaks: Gradual memory increase, eventual crash
• File locking: File access denied, sharing violations

2. Time-based Analysis

Correlate errors with usage patterns:

# Group errors by hour to find peak load issues
# Look for patterns related to:
- Daily sample processing batches
- Instrument data imports
- Report generation schedules
- Backup operations
- Maintenance windows

Proactive Monitoring Setup

1. Health Check URLs

Create monitoring endpoints:

• Database connectivity test page
• File system access test
• Integration service status
• Memory usage indicator
• Application version/build info

2. Log Rotation and Retention

Prevent disk space issues:

# Setup log cleanup job
forfiles /P "C:\inetpub\logs\LogFiles" /S /M *.log /D -30 /C "cmd /c del @path"

# Monitor disk space
wmic logicaldisk get size,freespace,caption

3. Alerting Thresholds

Set up monitoring for:

• Memory usage >80% for 5+ minutes
• Response times >30 seconds
• Error rate >5% in 10-minute window
• Disk space <10% remaining
• Failed database connections
• App pool recycles

Emergency Response Procedures

1. LIMS Application Down

Quick recovery steps:

1. Check app pool status and restart if needed

2. Verify database connectivity

3. Check recent Windows updates/patches

4. Review last 2 hours of logs for clues

5. Test with minimal user account

6. Check file system permissions

7. Verify certificate validity

2. Performance Degradation

Investigation sequence:

1. Check current memory/CPU usage

2. Analyze recent slow requests

3. Test database query performance

4. Check for file system issues

5. Review integration point status

6. Monitor active user sessions

3. Data Integrity Issues

When LIMS data seems corrupted:

1. Stop further data processing

2. Check database consistency

3. Review file upload logs

4. Verify backup integrity

5. Check integration data sources

6. Document timeline of events

Tools and Utilities Reference

Built-in Windows Tools

• Event Viewer: Application, System, Security logs
• Performance Monitor: Real-time counters and data collector sets
• Resource Monitor: Process and resource usage
• IIS Manager: Configuration and real-time monitoring
• Services Console: Service status and dependencies

Command Line Utilities

• appcmd: IIS configuration management
• logman: ETW trace management
• typeperf: Performance counter data collection
• netstat: Network connection monitoring
• tasklist/taskkill: Process management

Third-party Tools (if available)

• Process Monitor: File/registry access monitoring
• Wireshark: Network traffic analysis
• SQL Profiler: Database query monitoring
• LogParser: Advanced log file analysis