Play Video

Build your own Wix Rendering Platform – Gil Eckstein

Wix is mostly famous for its amazing drag & drop site building experience. While the Wix product & offerings has evolved a lot during the years, the site building capabilities remained our bread and butter. One of the core applications in Wix is the site renderer: a generic renderer for all Wix websites.

In this talk, Gil introduces us to the challenges of building a world class site rendering platform. We start from scratch, build our first basic site renderer, and gradually complicate things (AKA “platformize it”). Prepare yourself for an intense live coding session!

// The source code for the video: https://github.com/gileck/simple-site-renderer

For more engineering updates and insights:

* Follow us on Twitter → https://twitter.com/WixEng
* Subscribe our monthly newsletter → https://www.wix.engineering/subscribe
* Visit our blog → https://www.wix.engineering/blog
* Follow our Medium publication → https://medium.com/wix-engineering
Category

hey everyone my name is Gill I’m a front-end developer in weeks I joined weeks run two and a half years ago and in weeks I’m a part of the group that works on the site renderer so the site renderer is the application that renders all the week’s websites it renders around 50 million websites in a day and today we’re going to build a simple site renderer from scratch and to show you how our site renderer is implemented or how we and also to
demonstrate the challenges that we face while we are developing it I know that a site renderer application might sound like a unique application but you’ll be surprised to know that the challenges we face are actually very common and I’m sure that I’m sure that most of them will be relevant to you so like I said I’m from the site renderer group and in this group I’m a part of the platform team in the platform team we enable users to extend the
functionality of their week’s web sites using adding JavaScript code users can customize fully customize their websites so we run the code that our users write to their websites and we also build SDKs so so that using the SDK basically the code can interact with the weak sides and I’m going to show everything in the live demo by the way this feature is a part of a big product called week’s corvid it’s a set of developer tools on top of the Wix
ecosystem and if you want to know more about Wix corvid you can basically just just search for Wix Corbett to get more information so before we start the live demo I want to show you one of the most important products in weeks it’s the site editor the site editor is basically the place that the user create their website so I created a small simple site for this live demo so I added a button a text and an image and when I click publish a JSON
representation of the website is saved somewhere on the Internet let’s take a look at this JavaScript object so it includes it’s a very complicated object it includes everything that I added and edit in my website so I have here somewhere a component that’s called site button and if I’ll search for the label that I gave to my button it’ll it’s also somewhere here in this big object it’s basically a JavaScript object representing everything in my
website so this is basically the output of the editor and another output of the editor is the code that I can add to my website so in so in this so in the editor I can basically write any JavaScript code that I want and to fully customize my website so I added a simple line of code here obviously you can write any JavaScript code you want in this JavaScript code I basically changed the text of the text component and I access the text component
from the ID of the text component it says text 1 and on the button in the top right corner and that’s that’s how I access the component so this is another output of the editor once I publish my site this code is saved some somewhere on the internet so both the user code and also the JSON representation of the site both of them are the outputs of the editor and also inputs for the site renderer so the site renderer takes those two inputs and
render the website and that’s exactly what we’re going to to do also in our simple application our simple site render is going to take those two inputs render the components and also run this code and read and actually render this side so let’s see first of all this is this is the site the Wix website like you see we rent it renders the component and also ran the week’s code we changed that change the text component to it works and but the only
difference that we’re going to use a simple JSON representation like you see this object includes a lot of information about my website that we’re not going to use so instead of this object I created a simple a simple object let’s take a look at it so a simple array of components that I’m going to use that’s what I’m going to render it it’s basically array of objects each each object contains the structure of the component so this is the
structure of my website and let’s take a look of the others the fields that each come each component has a comp ID a type I have a button and image and a text it has the wicks code ID this is how I select the component using wicks code I’m going to talk about it later in the live demo each component has a data the the button has a label the image has a source and the text has a text and also a layout the X&Y exactly where I put it in the
editor and also the width and the height so this is the structure of the component that we’re going to use to render the site and another input this is one input of our application and the other input of our application it’s going to be just this week’s code that I’m going to run later in their live demo so from those two inputs we’re going to render the Wix website and again it’s obviously going to be a simple site renderer but you can imagine
that our site renderer is much more complex and handle much more of use cases but during during the live demo I’m going to talk about all the challenges that we face when we develop the site our site renderer so let’s get started first of all I’m running the application here serving I’m serving all the code that I’m writing here to in this window everybody can see this we’re going to use we’re going to use the console log for logging so make sure
you look in the top right corner is this big enough for everyone great so let’s get started or start so the first thing that I want to do is I want to fetch the components structure from the internet like I said when I click publish the editor saves the structure of the component somewhere in the internet and in the real world maybe I’ll have at the server will send me this URL but in our example and base it’s basically just going to be a URL
that’s served in my local host so I’m going to use fetch to get it and it’s going to be a JSON so I’m going to transform it to a JSON let’s see that I have my components here okay so I got my three components here and now I’m going to use them to render the site so now I’m the way that I’m going to render the components that to write the UI is using react I’m going to import react and I’m going to we’re going to implement a react component that
it’s going to render those component structures so we’re going to write implement a renderer component I haven’t implemented yet but once I’ll implement it I’ll pass it a component the components object the components array and I’m going to render it to the Dom so now we’re going to implement this renderer component it’s a real component that gets our components array and renders the component and I’m rendering rendering it into an empty div in
the Dom so let’s implement the renderer so this is the renderer code the renderer is a simple react component that takes the components array and now what I want to do I want to loop the component array and for each component structure I want to render a react component that that is corresponding to the react component to their components type so let’s see how I do it first of all I’m going to loop that I’m going to return here a div wrapping
Dave and now I’m going to loop the components array for each component structure I’m going to have a mapping between component type and a react component that we’re going to develop and so this this this object is going to give me a react component you just call it the you are components it’s going to get the key is going to be the comm type each component structure has a calm type and it’s going to give me a react component that I can render
here and I’m going to pass it the entire component structure so it can use it to render the component now let’s write the pure components object so it’s a JavaScript object it has the keys button image and text oops and a button is basically just a button and first of all it’s a react component that receives the props and renders a button and since I have the entire component structure here I can put the components the the props data label from
from the component structure for this button it’s going to be the label of the button let’s implement the image image is a simple image with the source is from props data source and text is basically gonna be a simple give props data text just before I’m gonna show you how it looks like I want to add the layout so I can have all the components in the right place so I’m going to wrap each component with a div and I’m gonna have some styling here
so for now I want to put each component on the exact place that the user put it in the editor so I’m going to use position:absolute for that the top is going to be the Y from the layout and the left’s going to be the eggs now now I’m going to use a position:absolute but maybe in the real world in a responsive website I’m going to have a grid here that based on the width of the screen but based on the width of the screen is going to it’s going to
decide where to exactly to put the component but in our simple scenario I’m just going to use position:absolute now I’m just want to put the width and the height of the components has to be on on the tag on the element tag itself so a little bit more styling the width and the height on the button and on the image and it doesn’t matter for that that deep it’s not going to matter because only the font size affects the text so I’m not gonna do it
okay do you think it will work let’s see what let’s see what we have here okay so it works I rendered the component and react basically tells me that I need to have a key for each give in a list so let’s add this something internal that reacts want me to do great so what I actually did here is I took the JSON the representation of the site and I basically implemented three via components the button the text in the image and I’m looping the
structure of the site and rendering the rear components in this in the exact spot in the exact spot that the user edited in the editor and now I basically have this JSON representation that I can change I can change the X&Y here or our other text here and it will represent in our website if i refresh it great any question about our simple renderer before we go on to more complicated things great so now after we finished the simple insight
rendering now I want to run wix code what I actually need to do I need to change run the weeks ago that will change the IM text – it works so exactly like the site structure also the weeks code is somewhere in the internet and I have a URL to get it and again in my case is just going to be a simple URL but in our site renderer it the server send the URL of the week’s code to the client so it’s gonna be weeks code and it’s instead of a JSON it’s
going to be a text let’s see that I have the Wix code it’s just it’s just a string and and so we got this week’s code string and now I want to execute this code so what I could do is I could execute it here but it might be an issue and let’s let’s to understand why it’s an issue let’s look at this week’s code this week’s code uses the SDK that I mentioned earlier and it’s the way that the user code basically the user code is interacting with the
Wix website and the components of the website but why do we even need this as decay users can can implement the same logic using the browser API or the window API so let’s see an example of that I could use I had here snippet great I could use a simple browser API instead of this SDK I get and get if I had if I put an ID of the component let’s add an ID of the component on the dome in the renderer I’m going to have an ID here with the comp ID so
if I had an ID so user users could use the window API to basically change the components if we did everything correct it and I’m just running this code which changes using the window API to change to change the component inner text so and I executed the code in the same context as the window so the user code can access the window and if we will do that so why do we need this SDK if we do that if we let users use the window API to change the
components of the website they will have a very intimate or very their implementation will be very coupled with our internal implementation their code will be coupled with our internal implementation and this will be very hard for us to make changes to how our application works if we for example decide to change the place of this ID to say maybe a tan wrapper Dom or to edit to change change this strength this will break user code so we don’t want
that coupling between users code and our internal implementation and it it gets even worse if users use classes that we use in our Dom so we don’t want users to use the window API to in their code we want to develop an SDK so they can so they can use this SDK we can change our internal implementation but keep the the API the same and then everything will just work but what if what if but we all know like ourselves we are all developers and we
know that if we have a chance to use something we know we really probably use it so if we still if even if we implement this SDK and still run the user code in the same context as the window people will still use the window API the browser API so we want to run the with the user code in an environment where the window is is not defined so they cannot use the window object to access access the site components so the way to do it is run the user
code in a way in a web worker web workers are are basically the way to run isolated code in the browser let’s play around with web workers I’m going to create a new worker here and the URL is going to be okay yes let’s see what we got here so I created a worker and I got I got an instant an instance of the worker it has a few api’s but I’m just going to use the post message API sorry which is basically to send a message to the worker and also
their own message API which lets me receive messages from the worker so the main thread and the worker are both communicating with their messages so I don’t I don’t remember if I said it but the worker is running in a main thread and sometimes in some browsers it’s different process and the browser promised the developer that they will not they will not share any global memories and in the web worker the window will not will not be available okay
so what I want to do now is I want to send the week’s code to the web worker and execute it in the web worker before I do that I want to play I’m going to just send a simple message to the web worker let’s try that hey worker and now this is going to be our worker code here in worker je s we did this worker Jaso busy and this is going to be a renderer so you know and just like in the main thread there is the window object that window global
object so also in the web worker of the self global object and all the api’s are all the postmessage api and also the own message API are on the self or on the self object so the own message this function receives a message I’m going to control Oh get and also send a post message to the worker Hey so also here I want to listen to messages from the worker also back in the renderer code okay so I send a message from the main thread to the web
worker and also I received the message in the web worker and also send the message back to the renderer so let’s see how it looks like so the message object is a complicated object but what I actually care about is the data of the OP of the message also here and here so let’s change it here change it here great so you can see that hi workers in worker jsn hi render is from them from the index.js where we write our main thread so now I want to
send weeks code the weeks go to the worker and execute it there so I’m sending this and in back in the worker j/s that’s printed here so in the worker Jess I have the week’s code so let’s first of all remove okay so let’s execute a week’s code here and let’s first of all see that it explodes that window is not defined so like you see window is not defined I don’t have access to the window in the web worker and let’s bring back this code and now
my code crashes because the O W is not defined so let’s look at this API so again this door W is the SDK that we give it’s a global object that we pass two weeks code that with the the user code uses to interact with the Wix website and also change the components so it’s a function that receives a selector of a component like we saw in the editor earlier and it returns an object that has a text field and when I set something to this field I need
to change that the data of the component so let’s see how we implement it so this toy W needs to be a global function to receive the selector and it returns an object let’s see first of all what I want to do before I return the object I want to find the relevant component from this selector so like we remember the selector is this ID for this it’s kind of like an ID for each component that helps that helps the user code to select a component in
their code and it’s stored in each component structure in the site structure so each component is in each component structure I have the week’s code ID so I want to find the right component the component that that for this selector so for to do that I want to have the components also in the webworkers so let’s send all the components also to the web worker I will put them globally so I can access them in the in the W function components great
great so now I want to find the right component from the components all right it’s called wicks code ID and it’s going to be a component let’s see that I should have here right I should have here the text component okay so I have here the text component and what I want to do now I need to return an object with filled with the text and what I actually want to do is I want to send a message to the back to the renderer and tell him to change the
component data that I have here that I have here so I’m going to use JavaScript setter to do that it’s going to be set text and basically this is a JavaScript feature that lets me write the code like this but it’s actually a function that receives this this this value and in the first step parameters so it’s going to be this new text and what I want to do now is I want to send a message to that back to the renderer to change this components text
to this new text so I’m gonna send a false message here let’s send the comp ID so I can have so it will be easy for me to find the component on the other side and I’m gonna send here a data object so you’ll be easy for me on the other side and the main thread just to texture this object and assign it to the original site structured data object so there’s gonna be text new text so now let’s implement in in the main thread the function that handles
this post message so I should have here that message is I’m printing so this is the message that I get in their main thread it’s get has a comp so let’s change it to comp ID which makes more sense comp ID and the new data that I need to assign to the old data so what I want to do now is again find the components based on the comp ID this needs to be a data so I’m finding from the components array I’m finding the component with that company from
the data let’s see that I found the text component so I found the text component and what I want to do now is I want to have to basically just object assign comp data the original data of the component and data data so let’s call it override date us so we won’t have to write data the data so i’ve override data here and now I’m just taking this new object and assign it to their old object we don’t need this here let’s clean our code we don’t need
this great we don’t need this great ok didn’t work any idea why didn’t work why didn’t we why didn’t let’s debug it a little bit sorry what do you mean we render so you’re right because because what happened here is the main thread rent we created a new worker that runs in a new thread and a lot of we send a message the worker gave back gave back a message but in the main thread we already rendered the components to the site and then when the
workers sent us this message we just basically just change an object and just nothing happened it was too late so will this fix it ugly ugly fix but let’s see if it works now it works because now we’re rendering the all the components after we get the message so but this is ugly and actually not what we want but what we do want is we want to separate this code from actually this function this code gets the week’s code creates creates the worker
create the worker send the message and receive message we want to take this code and put it here we’re gonna call it platform and it’s gonna do it’s gonna do all the platform code and and the contract between the platform function and the renderer is going to be that the platform we’re gonna call it here and we’re going to hope we going to wait wait for the platform to finish before we render the component so because we want the platform to run
weeks code to run the worker to load and the worker to send back the message to change a component and lo need only then we want to render the side components so to implement it will return a new promise here implement this code here and I’m going to need to add a sink here and I’m going to resolve the promise only after I change I change I assign the new components data great so components is I’m defining these components to be globally so I can
access it in the platform function great now it works and and this is actually a common pattern that we see in the site renderer that we want to do we want to run logic before we render the components we can maybe do stuff like load load dynamically load code of of react components based on whatever is in the pay or do do some layout logic here and do all of that before we render the components because once the components are rendered we don’t
want obviously to change them so this is a common pattern that we have here we could also wrap this code with an if is is user code easier the code do I have user code in this site and if not not even loaded the platform what if it’s just a simple site with no user code so I can do like this I can even I can even move this code out of this file and dynamically load it if we have platform if we have weeks ago so all of those all of those are
available to us because we made that separation of the renderer and the platform but in our case we’re just going to run the platform code great everybody understand following what I’m doing here great awesome okay so yeah yeah because that’s what the user wanted the user wanted to change this text which works before it shows the previous text he doesn’t want to have this flicker you know to change the alt text I mean that’s a part of the product
it depends what the user wants is if the user wants to wait for the it for if he wants to write his code so that I mean that we need to wait for for him before he renders the components and we’ll wait for it I mean really the user code returns a promise so he can do whatever he wants in the promise and once you resolve the promise we will render the components so we can go fetch data and then change the components it’s everything happens once you
resolve the promise then I didn’t obviously didn’t implement it here but that’s really how it works great so now I want to make things a little bit more complicated by adding more weeks code here and talk about that this will make us implement more advanced features in our site renderers so now I want to so right now what we’re doing was sending one message from the from the main thread to the worker and one message from the worker to it back to
the main thread and now I want to implement more code so I’m going to move to this new branch and now I added more code now adding code that I need to click when I click the button with the user click the button it changes the source of the image so to implement this code now I want to send more than one message from also from the main thread to the worker and also from the worker to the main thread so instead of having to handle these multiple
messages in the on message function of each also in the main thread and also in the worker I’m basically going to put a type on its each message a type that’s going to help me decide which function to run also in the main thread and also in there in the renderer so this is the only this is the difference between the the other branch and this branch let’s take a look at it for example in the worker I added a type set data in our in my message and
then in the main thread it’s easy for me to check this data in their own message simple code that takes the data of the message and call a function with the same type with the same type of a message you see I have a function here so it’s basically it’s just gonna help me right it’s going to be easy to add more types of messages now that I have this code so I did it also in the worker and also also in the main thread and also in the worker but the
same logic I keep the same logic from the last branch and I only add the types and logic that based on the type call a different function and also another small change I want to resolve the platform promise only after I finish executing wix code and not on the first message before that I resolved the promise once I get the first message now I want to finish executing with code and only then resolve their platform promise so to do that in the
worker after we evaluating the week’s code then I send the post message with a type worker done only after evaluating all the code I can do a lot of messages there and only after that send the post message worker done and in the main thread the worker done message has resolved the promise so this is all the changes between the last branch and the new branch we are the types 2 messages and we were resolving the platform promise after we finished
executing weeks code everybody is it clear the difference between the two branches great so we can continue so now I want to run this week’s code so now I’ve an error on click is not a function we need to implement the onclick function so let’s implement on click so it receives a callback and what I want to do now is I want to take this dysfunction this callback function and put it on the on click of the button so when the user will click the on
button I will call this function that’s what I want to do so let’s try to do it again I’m gonna follow the same pattern here with the ID and let’s go to the main thread oh I need to add a type here so let’s add a type to this message it’s going to be set event handler and let’s handle this message here it’s going to be a function the text company and callback and let’s find the component just like we did before now I have the components object
and since this components object is best to the react component if you remember in a renderer we passed we passed each of structure object to each react component so this props object that I have in this button react component is exactly the same object that I have here so I can basically put an on-click function I can basically put a callback on the on click function of the button and of the of the button structure and then in the props of the
button in the renderer I can add an onclick function here and from the props put on click or nothing if I don’t have a function so my code does not explode if I don’t have this onclick alright so a small recap of what we’ve done we have the onclick function we sent the function to the main thread and we put the callback on this object that is passed to the react component of the button which basically just put it on the onclick of the element
great what do you think will it work awesome great so we got an error and the error tells us that the post message is fail to execute because some function could not be cloned and if I look in the postmessage api i can see that i can only pass serialized data between the worker and and the main thread and also from the main thread to the worker only CLA serialized data is allowed in the post message function so I can not pass this function to the
main thread but what I can do is I can save this callback in the worker and under under some codec ID so I’m going to create a callback ID here we’re going to see in a second how we’re going to create it and then I’m going to have the object here and I’m going to save the callback in the worker under the callback ID great and only send the Kovac ID to the main thread and the Kovac ID is going to be a string that has the comp ID in it so I can
have multiple callbacks for multiple components and also the name of the event so I can have multiple events for for a component ID so this is going to be my callback I did that I was sent to the main thread and in the main thread instead of putting the callback in the unclick I’m going to put like a frog a proxy function which which is going to send a message back to the worker to trigger the real callback so it’s going to be something like this
gonna be worker post message the type is going to be handle event it’s going to get now instead of a callback it’s gonna be called a tidy I’m going to send it a callback ID and back in the worker the only thing that I want to change I want to add the title callback ID so let’s the handler let’s implement the handler here and event it’s a function that gets a Kobuk ID event handlers Kovach ID and I’m just gonna run the callback ID so let’s recap
so instead of sending the callback to the main thread I’m creating a callback ID saving the callback in the worker under the Quebec ID send the Kovac ID to the main thread and in the main thread and creating a proxy function that I’m going to put on the on click of the button that this callback this function is going to send the callback ID to the worker and the worker is going to trigger my function so if you are a back-end developer maybe it’s
kind of like an RPC and and let’s see if it works so I did not implement them on the source yet but let’s just put in constant it works it works great so we have the this working and now we want to implement the source and all we want to do is we already have all the infrastructure ready for for overriding data of components so it should be easy to set source I’m going to send a post message from type set data it’s going to get new source and new
source should work I have all the infrastructure already from the set X that we did earlier if I send a message actually like this it should change change SRC great this should be a source alright let’s see if it works it didn’t work anyone has any idea why didn’t work same same answer same question same answer great so let’s see what happened here back to the main thread so very similar to what I said before we rendering the component after
after the user code finish running we rendering the component then the user clicked the button we send a message to the worker which sent back a message to the main thread that changes some object the changes dish this object but nobody is aware about this change and we are not rerender in the components so what I could do here is I could call I could extract this function this code to a function and call it but but I don’t want to do it because
only one one component has changed and if there is a lot if there are a lot of components on this website this could be very slow to re-render the entire components every time that one component was changed so I want to rerender only this the image component if if that one if that one was changed and also to have this also again this coupling between the UI and the platform if I would call a render function each time each time as something here
change it would like it would couple that again the plot that platform code and the UI code that I actually want to separate and I have another I have another so I don’t want to do that but I have another another option is to put my state my components array on react state in the renderer and then use that state to set the new the new data of the component but that that will have the same performance issue as react will not be aware of exactly
one change and we’ll just rerender the entire components and also putting my state the entire the entire state on react state is also not a good idea in a big application I actually want to separate the UI from the logic and and keep my state in my logic part of the application so I don’t want to do that as well what I actually want to do is I want to use the state management to help me with my issue and so I will import more bigs and I will use
the observable function and I will wrap my state with the observe observable function I will wrap my in the entire components array so he’s gonna be components I’m going to call the observer observable function I’m going to I’m going to call the observable function with the components array and what that’s going to do but that’s gonna do now my components array will be an mobile object that it’s not exactly the components are right that we think
it is let’s look at so it’s basically a very strange object that what Mobius does it’s it looks every field and every and all the data of the of this object it transform it transform it into setters and getters so it will know when I access it and also when I change it so now I can tell Mobius what I wanted to do whenever something is changing what I wanted to do is I wanted to rerender the right component so I’m going to go to my render and
first of all something exploded because now this object is not realizable when I send it in a post message to the worker so I’m going to use the two j/s function to transform it back to regular objects before I send it to the worker now we don’t have an error okay so back to the renderer I want to tell Mobius I want to give mobic access to my react components so he could rerender the the react component when it was changed so I’m going to route
wrap I’m going to first of all imports observer from Mobius react and I’m going to wrap each react component with the observer function it’s going to return an array component this observer here and image so now instead of the real components that I implemented now I have react components that are wrapped with mobic react components and since each props object is also a mobile object Mobile’s can know that each field that each react component is
using it knows when it knows when it was changed so you can rerender the right component let’s see if it’s working great it’s working so basically I used Mobile’s to manage my state and help me rerender stuff when things changes and I don’t have I still have my separation of the platform code and also and also the UI code and that really helps me to have this clean code okay enough enough code for at least for now I want you a recap and tell you
what that the site render that we wrote is still a simple site renderer but the stuff that we talked about during during the live demo if it’s if it’s the rerender issue that we kept kept had having during during the live demo the separation between the UI code and the logic code and also stuff that we want to do before before we render because we can’t we cannot render and then do stuff is its challenges that we face and things that we talk
about all the time in our group and hopefully it was interesting for you I put the code in this github repository so if you want to take a look at this code you can check you can check it out here and you can even run it there and and play with it anything else I think we’re done thank you so much thank you so much [Music]