Testing Entity Framework context in-memory

I’m a great believer in having your tests as integrated and as “end-to-end” as possible, avoiding mocks on anything that does not cross a port or boundary, ala https://blogs.msdn.microsoft.com/ericgu/2014/12/01/unit-test-success-using-ports-adapters-and-simulators/

One thing I have always found was that this became tricky when it came to mocking out your data access, particularly with EF. You can quite easily abstract away your data access layer and that’s absolutely fine. However, I always found I lost something doing this, especially if a test encompasses that  more than one bit of database access – as you start dealing in fake data too often. Not to mention if you anything more funky than simple read/writes in the layer you’re abstracting then this isn’t going to be covered.

Entity Framework 7 or core are providing in memory versions of your collections which is great for this: https://prashantbrall.wordpress.com/2015/08/31/entity-framework-7-in-memory-testing/ so look into that if you’re on that version.

I am, however, still using EF6. Some third parties exists to do this, like Effort: https://github.com/tamasflamich/effort which works nicely. I like to have a wee bit more control over the mocked instance and have started creating my own library for mocking your context.

MockEF.<Framework>

https://github.com/MartinMilsom/MockEF

https://www.nuget.org/packages/MockEF.Moq/

https://www.nuget.org/packages/MockEF.Rhino/

I found I like to deal with the context in the same way I would any other mocked service/object etc and so I decided to create variations of the library for various popular mocking frameworks. To date, just Rhino.Mocks and Moq. Let’s say we are using MockEF.Rhino…  Then the library will use Rhino to create its mock of the context. I.e. An in-memory collection version of the context will be created by Rhino’s MockRepository, and thus any functions you like to use of Rhino still work – this also has the added benefit of being able to mock anything that exists on your context’s interface – so if you have a bunch of methods you’ve created – theses are just mocked in the same way you always would.

Getting Started

To set up your context mock, simply call:

var context = new ContextBuilder<IMyContext>()
      .Setup(x => x.Authors, new List<Author> { new Author { Id = 1, Name = "bob" }})
      .Setup(x => x.Books)
  .GetContext();

The type argument of Context builder must be an interface. Then to use this in your code, you can set up a context factory, like so:

public interface IFactory
{
  IMyContext Create();
}

public class Factory : IFactory
{
  public IMyFactory Create()
  {
    return new MyContext();
  }
}

//class that needs to use the context
public class Example
{
  private IFactory _factory;
  public Example(IFactory factory)
  {
    _factory = factory;
  }

  public bool MethodThatUsesContext()
  {
    //Important. Your context interface MUST implement IDisposable. 
    //Firstly because this won't work otherwise, Secondonly because - you should anyway.
    using (var context = _factory.Create())
    {
      //Use context here.
    }
    return true;
  }
}

//When testing you can then stub your factory to return the context you have set up, like so:
var factory = MockRepository.GenerateMock<IFactory>();
factory.Stub(x => x.Create()).Return(myMockedContext); //context built using Rhino.MockEF.
//then
var result = new Example(factory).MethodThatUsesContext();
Assert.IsTrue(result);


//When running the code not in test, in you DI registrations you can use something like:
container.Register<IFactory, Factory>();

Current Limitations

Currently, if you have any calls to .Find(..) this will only look for fields with the [PrimaryKey] attribute – if this is set up as something else or setup elsewhere -it’s likely to fail.

.SaveChanges() or .SaveChangesAsync() will pretty much do nothing. Any adds or updates are automatically applied to the in memory collection. Which is not the same behaviour as real world. A good example of this is that the Attach(..) and Add(..) methods do the same thing.

To be Done

Plan to add more mocking frameworks.

Extension methods on libraries. Rather than manually creating a ContextBuilder – hide the methods behind an extension applied to the mocking framework’s library.

Test support for versions < EF6

 

 

Keep an Eye on Your Memory Consumption When using Akka.Net

I was recently working on a system that essentially involved using Akka to repeatedly call an Api over HTTP, transform the data and store the result. Pretty simple stuff. Using Akka to do this seemed ideal as some of the data transforms were a little complex and I was dealing with a lot of data and a huge amounts of requests. I had setup the system to start its process every 5 minutes, using the built-in scheduler – like so:


system
.Scheduler
.Schedule(
TimeSpan.FromSeconds(0),
TimeSpan.FromMinutes(5),
someActor, someMessage);

What I did wrong

Now, to keep it simple, lets say I had three actor types; CollectDataActor, TransformDataActor and StoreDataActor – and the process simply involved the CollectDataActor calling the Api then telling the TransformDataActor to do its thing and in turn this tells the StoreDataActor to, well, save the data.

What I was doing to achieve this was calling:


var actor = Context.ActorOf(Context.Create<CollectDataActor>());
actor.Tell(MyMessage);

And this would then collect the data which came back as a List<> if data. And then essentially created a TransformDataActor for every item in this list, and told them to process it.

Why this is a problem

As I mentioned at the top of the post, I was scheduling this process to run every 5 minutes. This meant that every single time the process was scheduled to run new actors were created at each stage, a new actor to collect, a huge amount to transform and subsequently store. This resulted in memory consumption just increasing and increasing over time. Not good.

Fix

At first my solution to this was to kill all the actors I had after I was done with them. However, this became hard to manage as the system grew and it didn’t work at all well for re-sending dead messages – it was a hack, to be honest.

The solution I used was to create in all of the actors I’d need before I start running. I knew I’d need only one CollectDataActor per process run, I’d need a shed load of TransformDataActors and the same for SaveDataActors (1 for each data item I transform).

What I did was create a router with a round robin pool of the TransformDataActors and SaveDataActors. This is a feature of Akka.Net that will essentially keep a pool of any given actor type, and when you message the router it send to an Actor within the said pool which is not busy (or has the fewest numbers of letters in its mailbox). Then, rather than creating an actor each time from the CollectDataActor, I can just select the router by its path and send it.

Code:


//setup code.
var transformPops = Props.Create<TransformDataActors>().WithRouter(new RoundRobinPool(50));
var transformRouter = system.ActorOf(transformPops , "transformDataRouter");

var saveProps = Props.Create<SaveDataActors>().WithRouter(new RoundRobinPool(50));
var saveRouter = system.ActorOf(saveProps , "saveDataRouter");

With these set up, the CollectDataActor code can look a little something like:


public class CollectDataActor : ReceiveActor
{
    public CollectDataActor()
    {
         Receive<object>(message => Handle(message));
    }

    private void Handle(object message)
    {
        List<TypeMcType> data = CollectData();

        foreach (var item in data)
        {
           var actorPath = "akka://path/to/your/system/transformRouter";
           var actor = Context.ActorSelection(actorPath);
           actor.Tell(item);
           //NOTE: here we use "ActorSelection", we do not create a new actor
           // -  this is the key difference!
        }
    }
}

Authenticate to Mongo Database that isn’t “admin”

If you have set up your Mongo instance by adding a user to the “admin” database to authenticate against, you may of run into some confusion about how you connect & authenticate against another database on that server.

Let’s say you are trying to use the “local” database… If you using command line, you would need to add the parameter –authenticationDatabase. So your connection would look something like:

mongo myserver.com --username martin.milsom --authenticationDatabase admin -p myPassword

The extra parameter lets Mongo know where the user is to authenticate against.

Now, if you are using the C# driver for this, as I was, then the answer is to include the same parameter in the connection URL, for example:

var connectionString = "mongodb://martin.milsom:myPassword@myServer.com:27017/?authenticationDatabase=admin";
var database = new MongoClient(connectionString).GetDatabase("local");

Note here, the Url Parameter "?authenticationDatabase=admin" is how you add the parameter.

Retrieve assigned GUID from SQL insert

When inserting into a Sql table with a Guid assigned using the default of NEWID() you cannot use ‘SCOPE_TO_IDENTITY’ to return the Guid you’ve been assigned into you application (like you would to retrieve an integer ID). It doesn’t work for Guids because they’re not number values and thus cannot be an identity field. The answer was to retrieve the Guid using the following query, for a table that has a column name TableNameGuid and ColumnName:


INSERT INTO TableName (ColumnName)
OUTPUT inserted.TableNameGuid
VALUES (Value)

The key part to this is the OUTPUT, its what returns the value. Then in the C# code I would call something like


using (SqlCommand cmd = new SqlCommand(yourSqlQueryString, connection))
{
     connection.open();
     Guid assignedGuid = (Guid)cmdCheck.ExecuteScalar(); //gets the guid
     connection.Close();
}

Which assigns your new, freshly created, Guid to the variable assignedGuid.